User interface for managing controllable external devices

ABSTRACT

An electronic device, with a display, a touch-sensitive surface, one or more processors and memory, displays a first representation of a first controllable external device, where the first controllable external device is situated at a location. The device detects a first user input corresponding to a selection of the first representation of the first controllable external device. The device, after detecting the first user input, adds data identifying the first controllable external device and a first state of the first controllable external device in a scene profile.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/427,516, titled “USER INTERFACE FOR MANAGING CONTROLLABLE EXTERNALDEVICES,” filed on Feb. 8, 2017, which claims priority to U.S.provisional patent application 62/349,057, titled “USER INTERFACE FORMANAGING CONTROLLABLE EXTERNAL DEVICES,” filed on Jun. 12, 2016, thecontents of which are hereby incorporated by reference in theirentirety.

This application relates to U.S. patent application Ser. No. 14/725,912,titled “Accessory Management System Using Environment Model,” filed May28, 2015, the content of which is hereby incorporated by reference inits entirety. This application also relates to the U.S. ProvisionalPatent Application titled “Devices and Methods for Accessing PrevalentDevice Functions,” filed Jun. 12, 2016, the content of which is herebyincorporated by reference in its entirety.

FIELD

The present disclosure relates generally to computer user interfaces,and more specifically to techniques for managing controllable externaldevices.

BACKGROUND

Electronic devices are becoming increasingly popular in a range ofapplications. Mobile phones, tablet computers, home entertainmentsystems, and the like are just some of the electronic devices usersinteract with regularly.

In addition, various electronically controllable devices, such asthermostats, lighting devices, household appliances, etc. are alsobecoming more popular.

BRIEF SUMMARY

Some techniques for managing controllable external devices usingelectronic devices, however, are generally cumbersome and inefficient.For example, some existing techniques use a complex and time-consuminguser interface, which may include multiple key presses or keystrokes.Existing techniques require more time than necessary, wasting user timeand device energy. This latter consideration is particularly importantin battery-operated devices.

Accordingly, the present technique provides electronic devices withfaster, more efficient methods and interfaces for managing controllableexternal devices. Such methods and interfaces optionally complement orreplace other methods for managing controllable external devices. Suchmethods and interfaces reduce the cognitive burden on a user and producea more efficient human-machine interface. For battery-operated computingdevices, such methods and interfaces conserve power and increase thetime between battery charges.

A method includes: at an electronic device with a display, atouch-sensitive surface, one or more processors, and memory: displaying,on a first user interface screen, an indication of a location;displaying, on the first user interface screen, a representation of acontrollable external device at the location; detecting a first userinput corresponding to selection of the controllable external device; inresponse to detecting the first user input, displaying a user interfaceobject indicating a plurality of possible states of the controllableexternal device; while displaying the user interface object indicatingthe plurality of possible states of the controllable external device,detecting a second user input corresponding to selection of a designatedstate of the controllable external device; and in response to detectingthe second user input, sending instructions to set the controllableexternal device to the designated state.

An electronic device includes: a display; a touch-sensitive surface; oneor more processors; a memory; and one or more programs, where the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: displaying, on a first user interface screen, anindication of a location; displaying, on the first user interfacescreen, a representation of a controllable external device at thelocation; detecting a first user input corresponding to selection of thecontrollable external device; in response to detecting the first userinput, displaying a user interface object indicating a plurality ofpossible states of the controllable external device; while displayingthe user interface object indicating the plurality of possible states ofthe controllable external device, detecting a second user inputcorresponding to selection of a designated state of the controllableexternal device; and in response to detecting the second user input,sending instructions to set the controllable external device to thedesignated state.

A non-transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: display, ona first user interface screen, an indication of a location; display, onthe first user interface screen, a representation of a controllableexternal device at the location; detect a first user input correspondingto selection of the controllable external device; in response todetecting the first user input, display a user interface objectindicating a plurality of possible states of the controllable externaldevice; while displaying the user interface object indicating theplurality of possible states of the controllable external device, detecta second user input corresponding to selection of a designated state ofthe controllable external device; and in response to detecting thesecond user input, send instructions to set the controllable externaldevice to the designated state.

A transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: display, ona first user interface screen, an indication of a location; display, onthe first user interface screen, a representation of a controllableexternal device at the location; detect a first user input correspondingto selection of the controllable external device; in response todetecting the first user input, display a user interface objectindicating a plurality of possible states of the controllable externaldevice; while displaying the user interface object indicating theplurality of possible states of the controllable external device, detecta second user input corresponding to selection of a designated state ofthe controllable external device; and in response to detecting thesecond user input, send instructions to set the controllable externaldevice to the designated state.

An electronic device includes: a display; a touch-sensitive surface;means for displaying, on a first user interface screen, an indication ofa location; means for displaying, on the first user interface screen, arepresentation of a controllable external device at the location; meansfor detecting a first user input corresponding to selection of thecontrollable external device; means responsive to detecting the firstuser input for displaying a user interface object indicating a pluralityof possible states of the controllable external device; means for, whiledisplaying the user interface object indicating the plurality ofpossible states of the controllable external device, detecting a seconduser input corresponding to selection of a designated state of thecontrollable external device; and means responsive to detecting thesecond user input for sending instructions to set the controllableexternal device to the designated state.

An electronic device includes: a display unit configured to display agraphic user interface; a touch-sensitive surface unit configured toreceive contacts; and a processing unit coupled to the display unit andthe touch-sensitive surface unit, the processing unit configured to:enable display of, on a first user interface screen, an indication of alocation; enable display of, on the first user interface screen, arepresentation of a controllable external device at the location; detecta first user input corresponding to selection of the controllableexternal device; in response to detecting the first user input, enabledisplay of a user interface object indicating a plurality of possiblestates of the controllable external device; while the user interfaceobject indicating the plurality of possible states of the controllableexternal device is displayed, detect a second user input correspondingto selection of a designated state of the controllable external device;and in response to detecting the second user input, cause sending ofinstructions to set the controllable external device to the designatedstate.

A method includes: at an electronic device with a display, atouch-sensitive surface, one or more processors, and memory: displayinga first representation of a first controllable external device, wherethe first controllable external device is situated at a location;detecting a first user input corresponding to a selection of the firstcontrollable external device; and after detecting the first user input,including data identifying the first controllable external device and afirst state of the first controllable external device in a sceneprofile.

An electronic device includes: a display; a touch-sensitive surface; oneor more processors; a memory; and one or more programs, where the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: displaying a first representation of a firstcontrollable external device, where the first controllable externaldevice is situated at a location; detecting a first user inputcorresponding to a selection of the first controllable external device;and after detecting the first user input, including data identifying thefirst controllable external device and a first state of the firstcontrollable external device in a scene profile.

A non-transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: display afirst representation of a first controllable external device, where thefirst controllable external device is situated at a location; detect afirst user input corresponding to a selection of the first controllableexternal device; and after detecting the first user input, include dataidentifying the first controllable external device and a first state ofthe first controllable external device in a scene profile.

A transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: display afirst representation of a first controllable external device, where thefirst controllable external device is situated at a location; detect afirst user input corresponding to a selection of the first controllableexternal device; and after detecting the first user input, include dataidentifying the first controllable external device and a first state ofthe first controllable external device in a scene profile.

An electronic device includes: a display; a touch-sensitive surface;means for displaying a first representation of a first controllableexternal device, where the first controllable external device issituated at a location; means for detecting a first user inputcorresponding to a selection of the first controllable external device;and means for, after detecting the first user input, including dataidentifying the first controllable external device and a first state ofthe first controllable external device in a scene profile.

An electronic device includes: a display unit configured to display agraphic user interface; a touch-sensitive surface unit configured toreceive contacts; and a processing unit coupled to the display unit andthe touch-sensitive surface unit, the processing unit configured to:enable display of a first representation of a first controllableexternal device, where the first controllable external device issituated at a location; detect a first user input corresponding to aselection of the first controllable external device; and after detectingthe first user input, include data identifying the first controllableexternal device and a first state of the first controllable externaldevice in a scene profile.

A method includes: at an electronic device with a display, atouch-sensitive surface, one or more processors, and memory: detecting afirst user input corresponding to selection of a first criterion; inresponse to detecting the first user input, associating the firstcriterion with an automation profile, where the automation profileincludes data representing: automation criteria including at least thefirst criterion, and at least one designated state for at least onecontrollable external device situated at a location; detecting a seconduser input corresponding to selection of a first controllable externaldevice situated at the location; adding data indicating a designatedstate of the first controllable external device to the automationprofile; determining whether the automation criteria has been met; andin accordance with a determination that the automation criteria has beenmet, sending instructions to set the first controllable external deviceto the designated state.

An electronic device includes: a display; a touch-sensitive surface; oneor more processors; a memory; and one or more programs, where the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: detecting a first user input corresponding toselection of a first criterion; in response to detecting the first userinput, associating the first criterion with an automation profile, wherethe automation profile includes data representing: automation criteriaincluding at least the first criterion, and at least one designatedstate for at least one controllable external device situated at alocation; detecting a second user input corresponding to selection of afirst controllable external device situated at the location; adding dataindicating a designated state of the first controllable external deviceto the automation profile; determining whether the automation criteriahas been met; and in accordance with a determination that the automationcriteria has been met, sending instructions to set the firstcontrollable external device to the designated state.

A non-transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: detect afirst user input corresponding to selection of a first criterion; inresponse to detecting the first user input, associate the firstcriterion with an automation profile, where the automation profileincludes data representing: automation criteria including at least thefirst criterion, and at least one designated state for at least onecontrollable external device situated at a location; detect a seconduser input corresponding to selection of a first controllable externaldevice situated at the location; add data indicating a designated stateof the first controllable external device to the automation profile;determine whether the automation criteria has been met; and inaccordance with a determination that the automation criteria has beenmet, send instructions to set the first controllable external device tothe designated state.

A transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: detect afirst user input corresponding to selection of a first criterion; inresponse to detecting the first user input, associate the firstcriterion with an automation profile, where the automation profileincludes data representing: automation criteria including at least thefirst criterion, and at least one designated state for at least onecontrollable external device situated at a location; detect a seconduser input corresponding to selection of a first controllable externaldevice situated at the location; add data indicating a designated stateof the first controllable external device to the automation profile;determine whether the automation criteria has been met; and inaccordance with a determination that the automation criteria has beenmet, send instructions to set the first controllable external device tothe designated state.

An electronic device comprises: a display; a touch-sensitive surface;means for detecting a first user input corresponding to selection of afirst criterion; means responsive to detecting the first user input forassociating the first criterion with an automation profile, where theautomation profile includes data representing: automation criteriaincluding at least the first criterion, and at least one designatedstate for at least one controllable external device situated at alocation; means for detecting a second user input corresponding toselection of a first controllable external device situated at thelocation; means for adding data indicating a designated state of thefirst controllable external device to the automation profile; means fordetermining whether the automation criteria has been met; and means for,in accordance with a determination that the automation criteria has beenmet, sending instructions to set the first controllable external deviceto the designated state.

An electronic device includes: a display unit configured to display agraphic user interface; a touch-sensitive surface unit configured toreceive contacts; and a processing unit coupled to the display unit andthe touch-sensitive surface unit, the processing unit configured to:detect a first user input corresponding to selection of a firstcriterion; in response to detecting the first user input, associate thefirst criterion with an automation profile, where the automation profileincludes data representing: automation criteria including at least thefirst criterion, and at least one designated state for at least onecontrollable external device situated at a location; detect a seconduser input corresponding to selection of a first controllable externaldevice situated at the location; add data indicating a designated stateof the first controllable external device to the automation profile;determine whether the automation criteria has been met; and inaccordance with a determination that the automation criteria has beenmet, cause sending of instructions to set the first controllableexternal device to the designated state.

A method includes: at an electronic device with a display, atouch-sensitive surface, a camera, one or more processors, and memory:capturing, using the camera, an image of a pattern; determining that thepattern corresponds to a controllable external device, where thecontrollable external device is configured to operate in two or morestates and is remotely controllable between the two or more states; inresponse to a determination that the pattern corresponds to thecontrollable external device, associating the controllable externaldevice with a location, where the location has at least one designatedroom; detecting a first user input representing selection of a room fromthe at least one designated room; and in response to detecting the firstuser input, associating the selected room with the controllable externaldevice.

An electronic device includes: a display; a touch-sensitive surface; acamera; one or more processors; a memory; and one or more programs,where the one or more programs are stored in the memory and configuredto be executed by the one or more processors, the one or more programsincluding instructions for: capturing, using the camera, an image of apattern; determining that the pattern corresponds to a controllableexternal device, where the controllable external device is configured tooperate in two or more states and is remotely controllable between thetwo or more states; in response to a determination that the patterncorresponds to the controllable external device, associating thecontrollable external device with a location, where the location has atleast one designated room; detecting a first user input representingselection of a room from the at least one designated room; and inresponse to detecting the first user input, associating the selectedroom with the controllable external device.

A non-transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay, a touch-sensitive surface, and a camera, cause the device to:capture, using the camera, an image of a pattern; determine that thepattern corresponds to a controllable external device, where thecontrollable external device is configured to operate in two or morestates and is remotely controllable between the two or more states; inresponse to a determination that the pattern corresponds to thecontrollable external device, associate the controllable external devicewith a location, where the location has at least one designated room;detect a first user input representing selection of a room from the atleast one designated room; and in response to detecting the first userinput, associate the selected room with the controllable externaldevice.

A transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay, a touch-sensitive surface, and a camera, cause the device to:capture, using the camera, an image of a pattern; determine that thepattern corresponds to a controllable external device, where thecontrollable external device is configured to operate in two or morestates and is remotely controllable between the two or more states; inresponse to a determination that the pattern corresponds to thecontrollable external device, associate the controllable external devicewith a location, where the location has at least one designated room;detect a first user input representing selection of a room from the atleast one designated room; and in response to detecting the first userinput, associate the selected room with the controllable externaldevice.

An electronic device includes: a display; a touch-sensitive surface; acamera; means for capturing, using the camera, an image of a pattern;means for determining that the pattern corresponds to a controllableexternal device, where the controllable external device is configured tooperate in two or more states and is remotely controllable between thetwo or more states; means response to a determination that the patterncorresponds to the controllable external device for associating thecontrollable external device with a location, where the location has atleast one designated room; means for detecting a first user inputrepresenting selection of a room from the at least one designated room;and means responsive to detecting the first user input, associating theselected room with the controllable external device.

An electronic device includes: a display unit configured to display agraphic user interface; a touch-sensitive surface unit configured toreceive contacts; a camera unit configured to capture images; and aprocessing unit coupled to the display unit and the touch-sensitivesurface unit, the processing unit configured to: cause capturing, usingthe camera, an image of a pattern; determine that the patterncorresponds to a controllable external device, where the controllableexternal device is configured to operate in two or more states and isremotely controllable between the two or more states; in response to adetermination that the pattern corresponds to the controllable externaldevice, associate the controllable external device with a location,where the location has at least one designated room; detect a first userinput representing selection of a room from the at least one designatedroom; and in response to detecting the first user input, associate theselected room with the controllable external device.

A method includes: at an electronic device with a display, atouch-sensitive surface, one or more processors, and memory: detectinguser input corresponding to designation of location-based criteria,where the location-based criteria are based at least in part on thelocation of the electronic device with respect to a first locationhaving at least one controllable external device; in response todetecting the user input corresponding to designation of the criteria,adding the location-based criteria to an automation profile associatedwith the location, where the automation profile includes datarepresenting automation criteria for controlling the at least onecontrollable external device and a designated state of the at least onecontrollable external device, and where the automation criteria forcontrolling the at least one controllable external device includes thelocation-based criteria; determining the location of the electronicdevice; determining whether the automation criteria for controlling theat least one controllable external device is met based at least in parton the determined location of the electronic device; and in accordancewith a determination that the automation criteria for controlling the atleast one controllable external device is met, sending instructions toset the at least one controllable external device to the designatedstate.

An electronic device includes: a display; a touch-sensitive surface; oneor more processors; a memory; and one or more programs, where the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: detecting user input corresponding to designation oflocation-based criteria, where the location-based criteria are based atleast in part on the location of the electronic device with respect to afirst location having at least one controllable external device; inresponse to detecting the user input corresponding to designation of thecriteria, adding the location-based criteria to an automation profileassociated with the location, where the automation profile includes datarepresenting automation criteria for controlling the at least onecontrollable external device and a designated state of the at least onecontrollable external device, and where the automation criteria forcontrolling the at least one controllable external device includes thelocation-based criteria; determining the location of the electronicdevice; determining whether the automation criteria for controlling theat least one controllable external device is met based at least in parton the determined location of the electronic device; and in accordancewith a determination that the automation criteria for controlling the atleast one controllable external device is met, sending instructions toset the at least one controllable external device to the designatedstate.

A non-transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: detect userinput corresponding to designation of location-based criteria, where thelocation-based criteria are based at least in part on the location ofthe electronic device with respect to a first location having at leastone controllable external device; in response to detecting the userinput corresponding to designation of the criteria, add thelocation-based criteria to an automation profile associated with thelocation, where the automation profile includes data representingautomation criteria for controlling the at least one controllableexternal device and a designated state of the at least one controllableexternal device, and where the automation criteria for controlling theat least one controllable external device includes the location-basedcriteria; determine the location of the electronic device; determinewhether the automation criteria for controlling the at least onecontrollable external device is met based at least in part on thedetermined location of the electronic device; and in accordance with adetermination that the automation criteria for controlling the at leastone controllable external device is met, send instructions to set the atleast one controllable external device to the designated state.

A transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: detect userinput corresponding to designation of location-based criteria, where thelocation-based criteria are based at least in part on the location ofthe electronic device with respect to a first location having at leastone controllable external device; in response to detecting the userinput corresponding to designation of the criteria, add thelocation-based criteria to an automation profile associated with thelocation, where the automation profile includes data representingautomation criteria for controlling the at least one controllableexternal device and a designated state of the at least one controllableexternal device, and where the automation criteria for controlling theat least one controllable external device includes the location-basedcriteria; determine the location of the electronic device; determinewhether the automation criteria for controlling the at least onecontrollable external device is met based at least in part on thedetermined location of the electronic device; and in accordance with adetermination that the automation criteria for controlling the at leastone controllable external device is met, send instructions to set the atleast one controllable external device to the designated state.

An electronic device includes: a display; a touch-sensitive surface;means for detecting user input corresponding to designation oflocation-based criteria, where the location-based criteria are based atleast in part on the location of the electronic device with respect to afirst location having at least one controllable external device; meansresponsive to detecting the user input corresponding to designation ofthe criteria for adding the location-based criteria to an automationprofile associated with the location, where the automation profileincludes data representing automation criteria for controlling the atleast one controllable external device and a designated state of the atleast one controllable external device, and where the automationcriteria for controlling the at least one controllable external deviceincludes the location-based criteria; means for determining the locationof the electronic device; means for determining whether the automationcriteria for controlling the at least one controllable external deviceis met based at least in part on the determined location of theelectronic device; and means for, in accordance with a determinationthat the automation criteria for controlling the at least onecontrollable external device is met, sending instructions to set the atleast one controllable external device to the designated state.

An electronic device includes: a display unit configured to display agraphic user interface; a touch-sensitive surface unit configured toreceive contacts; and a processing unit coupled to the display unit andthe touch-sensitive surface unit, the processing unit configured to:detect user input corresponding to designation of location-basedcriteria, where the location-based criteria are based at least in parton the location of the electronic device with respect to a firstlocation having at least one controllable external device; in responseto detecting the user input corresponding to designation of thecriteria, add the location-based criteria to an automation profileassociated with the location, where the automation profile includes datarepresenting automation criteria for controlling the at least onecontrollable external device and a designated state of the at least onecontrollable external device, and where the automation criteria forcontrolling the at least one controllable external device includes thelocation-based criteria; determine the location of the electronicdevice; determine whether the automation criteria for controlling the atleast one controllable external device is met based at least in part onthe determined location of the electronic device; and in accordance witha determination that the automation criteria for controlling the atleast one controllable external device is met, cause sending ofinstructions to set the at least one controllable external device to thedesignated state.

A method includes: at an electronic device with a display, atouch-sensitive surface, one or more processors, and memory: determiningwhether criteria of an automation profile are met, where the automationprofile includes data representing criteria for controlling a state ofat least one controllable external device and designated states for theat least one controllable external device; in accordance with adetermination that the criteria of the automation profile are met,displaying a notification, where the notification includes an indicationthat identifies the automation profile and an indication to confirmimplementation of the automation profile; detecting a user input; and inresponse to detecting the user input: in accordance with the user inputcorresponding to confirmation of implementation of the automationprofile, sending instructions to implement the designated states of theautomation profile; and in accordance with the user input correspondingto non-confirmation of implementation of the automation profile, sendinginstructions not to implement the designated states of the automationprofile.

An electronic device includes: a display; a touch-sensitive surface; oneor more processors; a memory; and one or more programs, where the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: determining whether criteria of an automation profileare met, where the automation profile includes data representingcriteria for controlling a state of at least one controllable externaldevice and designated states for the at least one controllable externaldevice; in accordance with a determination that the criteria of theautomation profile are met, displaying a notification, where thenotification includes an indication that identifies the automationprofile and an indication to confirm implementation of the automationprofile; detecting a user input; and in response to detecting the userinput: in accordance with the user input corresponding to confirmationof implementation of the automation profile, sending instructions toimplement the designated states of the automation profile; and inaccordance with the user input corresponding to non-confirmation ofimplementation of the automation profile, sending instructions not toimplement the designated states of the automation profile.

A non-transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: determinewhether criteria of an automation profile are met, where the automationprofile includes data representing criteria for controlling a state ofat least one controllable external device and designated states for theat least one controllable external device; in accordance with adetermination that the criteria of the automation profile are met,display a notification, where the notification includes an indicationthat identifies the automation profile and an indication to confirmimplementation of the automation profile; detect a user input; and inresponse to detecting the user input: in accordance with the user inputcorresponding to confirmation of implementation of the automationprofile, send instructions to implement the designated states of theautomation profile; and in accordance with the user input correspondingto non-confirmation of implementation of the automation profile, sendinstructions not to implement the designated states of the automationprofile.

A transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: determinewhether criteria of an automation profile are met, where the automationprofile includes data representing criteria for controlling a state ofat least one controllable external device and designated states for theat least one controllable external device; in accordance with adetermination that the criteria of the automation profile are met,display a notification, where the notification includes an indicationthat identifies the automation profile and an indication to confirmimplementation of the automation profile; detect a user input; and inresponse to detecting the user input: in accordance with the user inputcorresponding to confirmation of implementation of the automationprofile, send instructions to implement the designated states of theautomation profile; and in accordance with the user input correspondingto non-confirmation of implementation of the automation profile, sendinstructions not to implement the designated states of the automationprofile.

An electronic device includes: a display; a touch-sensitive surface;means for determining whether criteria of an automation profile are met,where the automation profile includes data representing criteria forcontrolling a state of at least one controllable external device anddesignated states for the at least one controllable external device;means for, in accordance with a determination that the criteria of theautomation profile are met, displaying a notification, where thenotification includes an indication that identifies the automationprofile and an indication to confirm implementation of the automationprofile; means for detecting a user input; and means responsive todetecting the user input for: in accordance with the user inputcorresponding to confirmation of implementation of the automationprofile, sending instructions to implement the designated states of theautomation profile; and in accordance with the user input correspondingto non-confirmation of implementation of the automation profile, sendinginstructions not to implement the designated states of the automationprofile.

An electronic device includes: a display unit configured to display agraphic user interface; a touch-sensitive surface unit configured toreceive contacts; and a processing unit coupled to the display unit andthe touch-sensitive surface unit, the processing unit configured to:determine whether criteria of an automation profile are met, where theautomation profile includes data representing criteria for controlling astate of at least one controllable external device and designated statesfor the at least one controllable external device; in accordance with adetermination that the criteria of the automation profile are met,enable display of a notification, where the notification includes anindication that identifies the automation profile and an indication toconfirm implementation of the automation profile; detect a user input;and in response to detecting the user input: in accordance with the userinput corresponding to confirmation of implementation of the automationprofile, cause sending of instructions to implement the designatedstates of the automation profile; and in accordance with the user inputcorresponding to non-confirmation of implementation of the automationprofile, cause sending of instructions not to implement the designatedstates of the automation profile.

A method includes: at an electronic device with a display, atouch-sensitive surface, one or more processors, and memory: displayinga representation of a controllable external device, where thecontrollable external device is configured to operate in a plurality ofstates, and where the controllable external device is in a first stateof a plurality of states; detecting a first user input corresponding toa selection of the controllable external device; determining whether thefirst user input meets input criteria; in accordance with adetermination that the first user input does not meet the inputcriteria, sending instructions to set the state of the controllableexternal device to a second state of the plurality of states; and inaccordance with a determination that the first user input meets theinput criteria: displaying an affordance indicating the plurality ofstates of the controllable external device; while displaying theaffordance indicating the plurality of states of the controllableexternal device, detecting a second user input on the affordancecorresponding to selection of a third state of the plurality of states;and in response to detecting the second user input, sending instructionsto set the state of the controllable external device to the third state.

An electronic device includes: a display; a touch-sensitive surface; oneor more processors; a memory; and one or more programs, where the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: displaying a representation of a controllable externaldevice, where the controllable external device is configured to operatein a plurality of states, and where the controllable external device isin a first state of a plurality of states; detecting a first user inputcorresponding to a selection of the controllable external device;determining whether the first user input meets input criteria; inaccordance with a determination that the first user input does not meetthe input criteria, sending instructions to set the state of thecontrollable external device to a second state of the plurality ofstates; and in accordance with a determination that the first user inputmeets the input criteria: displaying an affordance indicating theplurality of states of the controllable external device; whiledisplaying the affordance indicating the plurality of states of thecontrollable external device, detecting a second user input on theaffordance corresponding to selection of a third state of the pluralityof states; and in response to detecting the second user input, sendinginstructions to set the state of the controllable external device to thethird state.

A non-transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: display arepresentation of a controllable external device, where the controllableexternal device is configured to operate in a plurality of states, andwhere the controllable external device is in a first state of aplurality of states; detect a first user input corresponding to aselection of the controllable external device; determine whether thefirst user input meets input criteria; in accordance with adetermination that the first user input does not meet the inputcriteria, send instructions to set the state of the controllableexternal device to a second state of the plurality of states; and inaccordance with a determination that the first user input meets theinput criteria: display an affordance indicating the plurality of statesof the controllable external device; while displaying the affordanceindicating the plurality of states of the controllable external device,detect a second user input on the affordance corresponding to selectionof a third state of the plurality of states; and in response todetecting the second user input, send instructions to set the state ofthe controllable external device to the third state.

A transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: display arepresentation of a controllable external device, where the controllableexternal device is configured to operate in a plurality of states, andwhere the controllable external device is in a first state of aplurality of states; detect a first user input corresponding to aselection of the controllable external device; determine whether thefirst user input meets input criteria; in accordance with adetermination that the first user input does not meet the inputcriteria, send instructions to set the state of the controllableexternal device to a second state of the plurality of states; and inaccordance with a determination that the first user input meets theinput criteria: display an affordance indicating the plurality of statesof the controllable external device; while displaying the affordanceindicating the plurality of states of the controllable external device,detect a second user input on the affordance corresponding to selectionof a third state of the plurality of states; and in response todetecting the second user input, send instructions to set the state ofthe controllable external device to the third state.

An electronic device includes: a display; a touch-sensitive surface;means for displaying a representation of a controllable external device,where the controllable external device is configured to operate in aplurality of states, and where the controllable external device is in afirst state of a plurality of states; means for detecting a first userinput corresponding to a selection of the controllable external device;means for determining whether the first user input meets input criteria;means for, in accordance with a determination that the first user inputdoes not meet the input criteria, sending instructions to set the stateof the controllable external device to a second state of the pluralityof states; and means for, in accordance with a determination that thefirst user input meets the input criteria: displaying an affordanceindicating the plurality of states of the controllable external device;while displaying the affordance indicating the plurality of states ofthe controllable external device, detecting a second user input on theaffordance corresponding to selection of a third state of the pluralityof states; and in response to detecting the second user input, sendinginstructions to set the state of the controllable external device to thethird state.

An electronic device includes: a display unit configured to display agraphic user interface; a touch-sensitive surface unit configured toreceive contacts; and a processing unit coupled to the display unit andthe touch-sensitive surface unit, the processing unit configured to:enable display of a representation of a controllable external device,where the controllable external device is configured to operate in aplurality of states, and where the controllable external device is in afirst state of a plurality of states; detect a first user inputcorresponding to a selection of the controllable external device;determine whether the first user input meets input criteria; inaccordance with a determination that the first user input does not meetthe input criteria, cause sending of instructions to set the state ofthe controllable external device to a second state of the plurality ofstates; and in accordance with a determination that the first user inputmeets the input criteria: enable display of an affordance indicating theplurality of states of the controllable external device; whiledisplaying the affordance indicating the plurality of states of thecontrollable external device, detect a second user input on theaffordance corresponding to selection of a third state of the pluralityof states; and in response to detecting the second user input, causesending of instructions to set the state of the controllable externaldevice to the third state.

A method includes: at an electronic device with a display, one or moreprocessors, and memory: determining a first position of the electronicdevice within a location, where the location includes at least onedesignated room; determining that the first position corresponds to afirst room of the at least one designated room; and in response todetermining that the first position corresponds to the first room,displaying a representation of a first controllable external deviceassociated with the first room.

An electronic device includes: a display; a touch-sensitive surface; oneor more processors; a memory; and one or more programs, where the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: determining a first position of the electronic devicewithin a location, where the location includes at least one designatedroom; determining that the first position corresponds to a first room ofthe at least one designated room; and in response to determining thatthe first position corresponds to the first room, displaying arepresentation of a first controllable external device associated withthe first room.

A non-transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay, cause the device to: determine a first position of theelectronic device within a location, where the location includes atleast one designated room; determine that the first position correspondsto a first room of the at least one designated room; and in response todetermining that the first position corresponds to the first room,display a representation of a first controllable external deviceassociated with the first room.

A transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay, cause the device to: determine a first position of theelectronic device within a location, where the location includes atleast one designated room; determine that the first position correspondsto a first room of the at least one designated room; and in response todetermining that the first position corresponds to the first room,display a representation of a first controllable external deviceassociated with the first room.

An electronic device includes: a display; means for determining a firstposition of the electronic device within a location, where the locationincludes at least one designated room; means for determining that thefirst position corresponds to a first room of the at least onedesignated room; and means responsive to determining that the firstposition corresponds to the first room for displaying a representationof a first controllable external device associated with the first room.

An electronic device includes: a display unit configured to display agraphic user interface; and a processing unit coupled to the displayunit and the touch-sensitive surface unit, the processing unitconfigured to: determine a first position of the electronic devicewithin a location, where the location includes at least one designatedroom; determine that the first position corresponds to a first room ofthe at least one designated room; and in response to determining thatthe first position corresponds to the first room, enable display of arepresentation of a first controllable external device associated withthe first room.

A method includes: at an electronic device with a display, atouch-sensitive surface, one or more processors, and memory: displayinga first affordance representing a first personal contact; detecting afirst user input corresponding to selection of the first affordancerepresenting the first personal contact; displaying a second affordancerepresenting an access parameter that determines access to a locationprofile associated with a location, where the location profile includesinformation regarding a controllable external device at the location;detecting a second user input corresponding to selection of the secondaffordance; granting an external device associated with the firstpersonal contact access to the location profile, where the access isbased on the access parameter; and sending data to the external deviceassociated with the first personal contact, the data including aninvitation to access the location profile.

An electronic device includes: a display; a touch-sensitive surface; oneor more processors; a memory; and one or more programs, where the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: displaying a first affordance representing a firstpersonal contact; detecting a first user input corresponding toselection of the first affordance representing the first personalcontact; displaying a second affordance representing an access parameterthat determines access to a location profile associated with a location,where the location profile includes information regarding a controllableexternal device at the location; detecting a second user inputcorresponding to selection of the second affordance; granting anexternal device associated with the first personal contact access to thelocation profile, where the access is based on the access parameter; andsending data to the external device associated with the first personalcontact, the data including an invitation to access the locationprofile.

A non-transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: display afirst affordance representing a first personal contact; detect a firstuser input corresponding to selection of the first affordancerepresenting the first personal contact; display a second affordancerepresenting an access parameter that determines access to a locationprofile associated with a location, where the location profile includesinformation regarding a controllable external device at the location;detect a second user input corresponding to selection of the secondaffordance; grant an external device associated with the first personalcontact access to the location profile, where the access is based on theaccess parameter; and send data to the external device associated withthe first personal contact, the data including an invitation to accessthe location profile.

A transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: display afirst affordance representing a first personal contact; detect a firstuser input corresponding to selection of the first affordancerepresenting the first personal contact; display a second affordancerepresenting an access parameter that determines access to a locationprofile associated with a location, where the location profile includesinformation regarding a controllable external device at the location;detect a second user input corresponding to selection of the secondaffordance; grant an external device associated with the first personalcontact access to the location profile, where the access is based on theaccess parameter; and send data to the external device associated withthe first personal contact, the data including an invitation to accessthe location profile.

An electronic device includes: a display; a touch-sensitive surface;means for displaying a first affordance representing a first personalcontact; means for detecting a first user input corresponding toselection of the first affordance representing the first personalcontact; means for displaying a second affordance representing an accessparameter that determines access to a location profile associated with alocation, where the location profile includes information regarding acontrollable external device at the location; means for detecting asecond user input corresponding to selection of the second affordance;means for granting an external device associated with the first personalcontact access to the location profile, where the access is based on theaccess parameter; and means for sending data to the external deviceassociated with the first personal contact, the data including aninvitation to access the location profile.

An electronic device includes: a display unit configured to display agraphic user interface; a touch-sensitive surface unit configured toreceive contacts; and a processing unit coupled to the display unit andthe touch-sensitive surface unit, the processing unit configured to:enable display of a first affordance representing a first personalcontact; detect a first user input corresponding to selection of thefirst affordance representing the first personal contact; enable displayof a second affordance representing an access parameter that determinesaccess to a location profile associated with a location, where thelocation profile includes information regarding a controllable externaldevice at the location; detect a second user input corresponding toselection of the second affordance; cause an external device associatedwith the first personal contact to be granted access to the locationprofile, where the access is based on the access parameter; and causesending data to the external device associated with the first personalcontact, the data including an invitation to access the locationprofile.

A method includes: at an electronic device with a display, atouch-sensitive surface, one or more processors, and memory: receivingdata including information related to a location, where the dataincludes information related to a first controllable external device atthe location; in accordance with receiving the data, displaying anaffordance corresponding to the first controllable external device;detecting a first user input corresponding to selection of theaffordance; and in response to detecting the first user input,displaying a user interface screen including an indication of thelocation and a representation of the first controllable external device.

An electronic device includes: a display; a touch-sensitive surface; oneor more processors; a memory; and one or more programs, where the one ormore programs are stored in the memory and configured to be executed bythe one or more processors, the one or more programs includinginstructions for: receiving data including information related to alocation, where the data includes information related to a firstcontrollable external device at the location; in accordance withreceiving the data, displaying an affordance corresponding to the firstcontrollable external device; detecting a first user input correspondingto selection of the affordance; and in response to detecting the firstuser input, displaying a user interface screen including an indicationof the location and a representation of the first controllable externaldevice.

A non-transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: receive dataincluding information related to a location, where the data includesinformation related to a first controllable external device at thelocation; in accordance with receiving the data, display an affordancecorresponding to the first controllable external device; detect a firstuser input corresponding to selection of the affordance; and in responseto detecting the first user input, display a user interface screenincluding an indication of the location and a representation of thefirst controllable external device.

A transitory computer-readable storage medium stores one or moreprograms, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device with adisplay and a touch-sensitive surface, cause the device to: receive dataincluding information related to a location, where the data includesinformation related to a first controllable external device at thelocation; in accordance with receiving the data, display an affordancecorresponding to the first controllable external device; detect a firstuser input corresponding to selection of the affordance; and in responseto detecting the first user input, display a user interface screenincluding an indication of the location and a representation of thefirst controllable external device.

An electronic device includes: a display; a touch-sensitive surface;means for receiving data including information related to a location,where the data includes information related to a first controllableexternal device at the location; means for, in accordance with receivingthe data, displaying an affordance corresponding to the firstcontrollable external device; means for detecting a first user inputcorresponding to selection of the affordance; and means responsive todetecting the first user input for displaying a user interface screenincluding an indication of the location and a representation of thefirst controllable external device.

An electronic device includes: a display unit configured to display agraphic user interface; a touch-sensitive surface unit configured toreceive contacts; and a processing unit coupled to the display unit andthe touch-sensitive surface unit, the processing unit configured to:receive data including information related to a location, where the dataincludes information related to a first controllable external device atthe location; in accordance with receiving the data, display anaffordance corresponding to the first controllable external device;detect a first user input corresponding to selection of the affordance;and in response to detecting the first user input, display a userinterface screen including an indication of the location and arepresentation of the first controllable external device.

Executable instructions for performing these functions are, optionally,included in a non-transitory computer-readable storage medium or othercomputer program product configured for execution by one or moreprocessors. Executable instructions for performing these functions are,optionally, included in a transitory computer-readable storage medium orother computer program product configured for execution by one or moreprocessors.

Thus, devices are provided with faster, more efficient methods andinterfaces for managing controllable external devices, therebyincreasing the effectiveness, efficiency, and user satisfaction withsuch devices. Such methods and interfaces may complement or replaceother methods for managing controllable external devices.

DESCRIPTION OF THE FIGURES

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1A is a block diagram illustrating a portable multifunction devicewith a touch-sensitive display in accordance with some embodiments.

FIG. 1B is a block diagram illustrating exemplary components for eventhandling in accordance with some embodiments.

FIG. 2 illustrates a portable multifunction device having a touch screenin accordance with some embodiments.

FIG. 3 is a block diagram of an exemplary multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments.

FIG. 4A illustrates an exemplary user interface for a menu ofapplications on a portable multifunction device in accordance with someembodiments.

FIG. 4B illustrates an exemplary user interface for a multifunctiondevice with a touch-sensitive surface that is separate from the displayin accordance with some embodiments.

FIG. 5A illustrates a personal electronic device in accordance with someembodiments.

FIG. 5B is a block diagram illustrating a personal electronic device inaccordance with some embodiments.

FIGS. 5C-5D illustrate exemplary components of a personal electronicdevice having a touch-sensitive display and intensity sensors inaccordance with some embodiments.

FIGS. 5E-5H illustrate exemplary components and user interfaces of apersonal electronic device in accordance with some embodiments.

FIGS. 6A-6S illustrate exemplary user interfaces in accordance with someembodiments.

FIGS. 7A-7C are flow diagrams illustrating a process for managingcontrollable external devices in accordance with some embodiments.

FIG. 8 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 9A-9F illustrate exemplary user interfaces in accordance with someembodiments.

FIG. 10 is a flow diagram illustrating a process for managingcontrollable external devices in accordance with some embodiments.

FIG. 11 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 12A-12C illustrate exemplary user interfaces in accordance withsome embodiments.

FIG. 13 is a flow diagram illustrating a process for managingcontrollable external devices in accordance with some embodiments.

FIG. 14 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 15A-15B illustrate exemplary user interfaces in accordance withsome embodiments.

FIG. 16 is a flow diagram illustrating a process for managingcontrollable external devices in accordance with some embodiments.

FIG. 17 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 18A-18C illustrate exemplary user interfaces in accordance withsome embodiments.

FIG. 19 is a flow diagram illustrating a process for managingcontrollable external devices in accordance with some embodiments.

FIG. 20 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 21 is a flow diagram illustrating a process for managingcontrollable external devices in accordance with some embodiments.

FIG. 22 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 23 is a flow diagram illustrating a process for managingcontrollable external devices in accordance with some embodiments.

FIG. 24 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIG. 25 illustrates an exemplary user interface in accordance with someembodiments.

FIG. 26 is a flow diagram illustrating a process for managingcontrollable external devices in accordance with some embodiments.

FIG. 27 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 28A-28E illustrate exemplary user interfaces in accordance withsome embodiments.

FIG. 29 is a flow diagram illustrating a process for managingcontrollable external devices in accordance with some embodiments.

FIG. 30 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 31A-31B illustrate exemplary user interfaces in accordance withsome embodiments.

FIG. 32 is a flow diagram illustrating a process for managingcontrollable external devices in accordance with some embodiments.

FIG. 33 is a functional block diagram of an electronic device inaccordance with some embodiments.

FIGS. 34A-34K illustrate exemplary user interfaces in accordance withsome embodiments.

DESCRIPTION OF EMBODIMENTS

The following description sets forth exemplary methods, parameters, andthe like. It should be recognized, however, that such description is notintended as a limitation on the scope of the present disclosure but isinstead provided as a description of exemplary embodiments.

There is a need for electronic devices that provide efficient methodsand interfaces for managing controllable external devices. In someembodiments, an electronic device provides access to designated (e.g.,favorite) controllable devices on a home screen for managing a system ofcontrollable devices. In some embodiments, a user can coarsely control adevice by toggling the state between two states (e.g., ON/OFF) usinginput with one type of characteristic and obtain more detailed controlof the device using input with a different characteristic. In someembodiments, a user-defined collection of devices is controlled inresponse to user-defined conditions. Various other techniques are alsodescribed. Such techniques can reduce the cognitive burden on a user whomanages controllable external devices, thereby enhancing productivity.Further, such techniques can reduce processor and battery powerotherwise wasted on redundant user inputs.

Below, FIGS. 1A-1B, 2, 3, 4A-4B, and 5A-5H provide a description ofexemplary devices for performing the techniques for managingcontrollable external devices. FIGS. 6A-6S, 9A-9F, 12A-12C, 15A-15B,18A-18C, 25, 28A-28E, 31A-31B, and 34A-34K illustrate exemplary userinterfaces for managing controllable external devices. FIGS. 7A-7C, 10,13, 16, 19, 21, 23, 26, 29, and 32 are flow diagrams illustratingmethods of managing controllable external devices in accordance withsome embodiments. The user interfaces in FIGS. 6A-6S, 9A-9F, 12A-12C,15A-15B, 18A-18C, 25, 28A-28E, 31A-31B, and 34A-34K are used toillustrate the processes described below, including the processes inFIGS. 7A-7C, 10, 13, 16, 19, 21, 23, 26, 29, and 32.

Although the following description uses terms “first,” “second,” etc. todescribe various elements, these elements should not be limited by theterms. These terms are only used to distinguish one element fromanother. For example, a first touch could be termed a second touch, and,similarly, a second touch could be termed a first touch, withoutdeparting from the scope of the various described embodiments. The firsttouch and the second touch are both touches, but they are not the sametouch.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

The term “if” is, optionally, construed to mean “when” or “upon” or “inresponse to determining” or “in response to detecting,” depending on thecontext. Similarly, the phrase “if it is determined” or “if [a statedcondition or event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event],” depending on the context.

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as PDA and/ormusic player functions. Exemplary embodiments of portable multifunctiondevices include, without limitation, the iPhone®, iPod Touch®, and iPad®devices from Apple Inc. of Cupertino, Calif. Other portable electronicdevices, such as laptops or tablet computers with touch-sensitivesurfaces (e.g., touch screen displays and/or touchpads), are,optionally, used. It should also be understood that, in someembodiments, the device is not a portable communications device, but isa desktop computer with a touch-sensitive surface (e.g., a touch screendisplay and/or a touchpad).

In the discussion that follows, an electronic device that includes adisplay and a touch-sensitive surface is described. It should beunderstood, however, that the electronic device optionally includes oneor more other physical user-interface devices, such as a physicalkeyboard, a mouse, and/or a joystick.

The device typically supports a variety of applications, such as one ormore of the following: a drawing application, a presentationapplication, a word processing application, a website creationapplication, a disk authoring application, a spreadsheet application, agaming application, a telephone application, a video conferencingapplication, an e-mail application, an instant messaging application, aworkout support application, a photo management application, a digitalcamera application, a digital video camera application, a web browsingapplication, a digital music player application, and/or a digital videoplayer application.

The various applications that are executed on the device optionally useat least one common physical user-interface device, such as thetouch-sensitive surface. One or more functions of the touch-sensitivesurface as well as corresponding information displayed on the deviceare, optionally, adjusted and/or varied from one application to the nextand/or within a respective application. In this way, a common physicalarchitecture (such as the touch-sensitive surface) of the deviceoptionally supports the variety of applications with user interfacesthat are intuitive and transparent to the user.

Attention is now directed toward embodiments of portable devices withtouch-sensitive displays. FIG. 1A is a block diagram illustratingportable multifunction device 100 with touch-sensitive display system112 in accordance with some embodiments. Touch-sensitive display 112 issometimes called a “touch screen” for convenience and is sometimes knownas or called a “touch-sensitive display system.” Device 100 includesmemory 102 (which optionally includes one or more computer-readablestorage mediums), memory controller 122, one or more processing units(CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry110, speaker 111, microphone 113, input/output (I/O) subsystem 106,other input control devices 116, and external port 124. Device 100optionally includes one or more optical sensors 164. Device 100optionally includes one or more contact intensity sensors 165 fordetecting intensity of contacts on device 100 (e.g., a touch-sensitivesurface such as touch-sensitive display system 112 of device 100).Device 100 optionally includes one or more tactile output generators 167for generating tactile outputs on device 100 (e.g., generating tactileoutputs on a touch-sensitive surface such as touch-sensitive displaysystem 112 of device 100 or touchpad 355 of device 300). Thesecomponents optionally communicate over one or more communication busesor signal lines 103.

As used in the specification and claims, the term “intensity” of acontact on a touch-sensitive surface refers to the force or pressure(force per unit area) of a contact (e.g., a finger contact) on thetouch-sensitive surface, or to a substitute (proxy) for the force orpressure of a contact on the touch-sensitive surface. The intensity of acontact has a range of values that includes at least four distinctvalues and more typically includes hundreds of distinct values (e.g., atleast 256). Intensity of a contact is, optionally, determined (ormeasured) using various approaches and various sensors or combinationsof sensors. For example, one or more force sensors underneath oradjacent to the touch-sensitive surface are, optionally, used to measureforce at various points on the touch-sensitive surface. In someimplementations, force measurements from multiple force sensors arecombined (e.g., a weighted average) to determine an estimated force of acontact. Similarly, a pressure-sensitive tip of a stylus is, optionally,used to determine a pressure of the stylus on the touch-sensitivesurface. Alternatively, the size of the contact area detected on thetouch-sensitive surface and/or changes thereto, the capacitance of thetouch-sensitive surface proximate to the contact and/or changes thereto,and/or the resistance of the touch-sensitive surface proximate to thecontact and/or changes thereto are, optionally, used as a substitute forthe force or pressure of the contact on the touch-sensitive surface. Insome implementations, the substitute measurements for contact force orpressure are used directly to determine whether an intensity thresholdhas been exceeded (e.g., the intensity threshold is described in unitscorresponding to the substitute measurements). In some implementations,the substitute measurements for contact force or pressure are convertedto an estimated force or pressure, and the estimated force or pressureis used to determine whether an intensity threshold has been exceeded(e.g., the intensity threshold is a pressure threshold measured in unitsof pressure). Using the intensity of a contact as an attribute of a userinput allows for user access to additional device functionality that mayotherwise not be accessible by the user on a reduced-size device withlimited real estate for displaying affordances (e.g., on atouch-sensitive display) and/or receiving user input (e.g., via atouch-sensitive display, a touch-sensitive surface, or aphysical/mechanical control such as a knob or a button).

As used in the specification and claims, the term “tactile output”refers to physical displacement of a device relative to a previousposition of the device, physical displacement of a component (e.g., atouch-sensitive surface) of a device relative to another component(e.g., housing) of the device, or displacement of the component relativeto a center of mass of the device that will be detected by a user withthe user's sense of touch. For example, in situations where the deviceor the component of the device is in contact with a surface of a userthat is sensitive to touch (e.g., a finger, palm, or other part of auser's hand), the tactile output generated by the physical displacementwill be interpreted by the user as a tactile sensation corresponding toa perceived change in physical characteristics of the device or thecomponent of the device. For example, movement of a touch-sensitivesurface (e.g., a touch-sensitive display or trackpad) is, optionally,interpreted by the user as a “down click” or “up click” of a physicalactuator button. In some cases, a user will feel a tactile sensationsuch as an “down click” or “up click” even when there is no movement ofa physical actuator button associated with the touch-sensitive surfacethat is physically pressed (e.g., displaced) by the user's movements. Asanother example, movement of the touch-sensitive surface is, optionally,interpreted or sensed by the user as “roughness” of the touch-sensitivesurface, even when there is no change in smoothness of thetouch-sensitive surface. While such interpretations of touch by a userwill be subject to the individualized sensory perceptions of the user,there are many sensory perceptions of touch that are common to a largemajority of users. Thus, when a tactile output is described ascorresponding to a particular sensory perception of a user (e.g., an “upclick,” a “down click,” “roughness”), unless otherwise stated, thegenerated tactile output corresponds to physical displacement of thedevice or a component thereof that will generate the described sensoryperception for a typical (or average) user.

It should be appreciated that device 100 is only one example of aportable multifunction device, and that device 100 optionally has moreor fewer components than shown, optionally combines two or morecomponents, or optionally has a different configuration or arrangementof the components. The various components shown in FIG. 1A areimplemented in hardware, software, or a combination of both hardware andsoftware, including one or more signal processing and/orapplication-specific integrated circuits.

Memory 102 optionally includes high-speed random access memory andoptionally also includes non-volatile memory, such as one or moremagnetic disk storage devices, flash memory devices, or othernon-volatile solid-state memory devices. Memory controller 122optionally controls access to memory 102 by other components of device100.

Peripherals interface 118 can be used to couple input and outputperipherals of the device to CPU 120 and memory 102. The one or moreprocessors 120 run or execute various software programs and/or sets ofinstructions stored in memory 102 to perform various functions fordevice 100 and to process data. In some embodiments, peripheralsinterface 118, CPU 120, and memory controller 122 are, optionally,implemented on a single chip, such as chip 104. In some otherembodiments, they are, optionally, implemented on separate chips.

RF (radio frequency) circuitry 108 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 108 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 108 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 108 optionally communicates with networks, such as theInternet, also referred to as the World Wide Web (WWW), an intranetand/or a wireless network, such as a cellular telephone network, awireless local area network (LAN) and/or a metropolitan area network(MAN), and other devices by wireless communication. The RF circuitry 108optionally includes well-known circuitry for detecting near fieldcommunication (NFC) fields, such as by a short-range communicationradio. The wireless communication optionally uses any of a plurality ofcommunications standards, protocols, and technologies, including but notlimited to Global System for Mobile Communications (GSM), Enhanced DataGSM Environment (EDGE), high-speed downlink packet access (HSDPA),high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO),HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), nearfield communication (NFC), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Bluetooth Low Energy (BTLE), Wireless Fidelity(Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n,and/or IEEE 802.11ac), voice over Internet Protocol (VoIP), Wi-MAX, aprotocol for e-mail (e.g., Internet message access protocol (IMAP)and/or post office protocol (POP)), instant messaging (e.g., extensiblemessaging and presence protocol (XMPP), Session Initiation Protocol forInstant Messaging and Presence Leveraging Extensions (SIMPLE), InstantMessaging and Presence Service (IMPS)), and/or Short Message Service(SMS), or any other suitable communication protocol, includingcommunication protocols not yet developed as of the filing date of thisdocument.

Audio circuitry 110, speaker 111, and microphone 113 provide an audiointerface between a user and device 100. Audio circuitry 110 receivesaudio data from peripherals interface 118, converts the audio data to anelectrical signal, and transmits the electrical signal to speaker 111.Speaker 111 converts the electrical signal to human-audible sound waves.Audio circuitry 110 also receives electrical signals converted bymicrophone 113 from sound waves. Audio circuitry 110 converts theelectrical signal to audio data and transmits the audio data toperipherals interface 118 for processing. Audio data is, optionally,retrieved from and/or transmitted to memory 102 and/or RF circuitry 108by peripherals interface 118. In some embodiments, audio circuitry 110also includes a headset jack (e.g., 212, FIG. 2). The headset jackprovides an interface between audio circuitry 110 and removable audioinput/output peripherals, such as output-only headphones or a headsetwith both output (e.g., a headphone for one or both ears) and input(e.g., a microphone).

I/O subsystem 106 couples input/output peripherals on device 100, suchas touch screen 112 and other input control devices 116, to peripheralsinterface 118. I/O subsystem 106 optionally includes display controller156, optical sensor controller 158, intensity sensor controller 159,haptic feedback controller 161, and one or more input controllers 160for other input or control devices. The one or more input controllers160 receive/send electrical signals from/to other input control devices116. The other input control devices 116 optionally include physicalbuttons (e.g., push buttons, rocker buttons, etc.), dials, sliderswitches, joysticks, click wheels, and so forth. In some alternateembodiments, input controller(s) 160 are, optionally, coupled to any (ornone) of the following: a keyboard, an infrared port, a USB port, and apointer device such as a mouse. The one or more buttons (e.g., 208, FIG.2) optionally include an up/down button for volume control of speaker111 and/or microphone 113. The one or more buttons optionally include apush button (e.g., 206, FIG. 2).

A quick press of the push button optionally disengages a lock of touchscreen 112 or optionally begins a process that uses gestures on thetouch screen to unlock the device, as described in U.S. patentapplication Ser. No. 11/322,549, “Unlocking a Device by PerformingGestures on an Unlock Image,” filed Dec. 23, 2005, U.S. Pat. No.7,657,849, which is hereby incorporated by reference in its entirety. Alonger press of the push button (e.g., 206) optionally turns power todevice 100 on or off. The functionality of one or more of the buttonsare, optionally, user-customizable. Touch screen 112 is used toimplement virtual or soft buttons and one or more soft keyboards.

Touch-sensitive display 112 provides an input interface and an outputinterface between the device and a user. Display controller 156 receivesand/or sends electrical signals from/to touch screen 112. Touch screen112 displays visual output to the user. The visual output optionallyincludes graphics, text, icons, video, and any combination thereof(collectively termed “graphics”). In some embodiments, some or all ofthe visual output optionally corresponds to user-interface objects.

Touch screen 112 has a touch-sensitive surface, sensor, or set ofsensors that accepts input from the user based on haptic and/or tactilecontact. Touch screen 112 and display controller 156 (along with anyassociated modules and/or sets of instructions in memory 102) detectcontact (and any movement or breaking of the contact) on touch screen112 and convert the detected contact into interaction withuser-interface objects (e.g., one or more soft keys, icons, web pages,or images) that are displayed on touch screen 112. In an exemplaryembodiment, a point of contact between touch screen 112 and the usercorresponds to a finger of the user.

Touch screen 112 optionally uses LCD (liquid crystal display)technology, LPD (light emitting polymer display) technology, or LED(light emitting diode) technology, although other display technologiesare used in other embodiments. Touch screen 112 and display controller156 optionally detect contact and any movement or breaking thereof usingany of a plurality of touch sensing technologies now known or laterdeveloped, including but not limited to capacitive, resistive, infrared,and surface acoustic wave technologies, as well as other proximitysensor arrays or other elements for determining one or more points ofcontact with touch screen 112. In an exemplary embodiment, projectedmutual capacitance sensing technology is used, such as that found in theiPhone® and iPod Touch® from Apple Inc. of Cupertino, Calif.

A touch-sensitive display in some embodiments of touch screen 112 is,optionally, analogous to the multi-touch sensitive touchpads describedin the following U.S. Pat. No. 6,323,846 (Westerman et al.), U.S. Pat.No. 6,570,557 (Westerman et al.), and/or U.S. Pat. No. 6,677,932(Westerman), and/or U.S. Patent Publication 2002/0015024A1, each ofwhich is hereby incorporated by reference in its entirety. However,touch screen 112 displays visual output from device 100, whereastouch-sensitive touchpads do not provide visual output.

A touch-sensitive display in some embodiments of touch screen 112 isdescribed in the following applications: (1) U.S. patent applicationSer. No. 11/381,313, “Multipoint Touch Surface Controller,” filed May 2,2006; (2) U.S. patent application Ser. No. 10/840,862, “MultipointTouchscreen,” filed May 6, 2004; (3) U.S. patent application Ser. No.10/903,964, “Gestures For Touch Sensitive Input Devices,” filed Jul. 30,2004; (4) U.S. patent application Ser. No. 11/048,264, “Gestures ForTouch Sensitive Input Devices,” filed Jan. 31, 2005; (5) U.S. patentapplication Ser. No. 11/038,590, “Mode-Based Graphical User InterfacesFor Touch Sensitive Input Devices,” filed Jan. 18, 2005; (6) U.S. patentapplication Ser. No. 11/228,758, “Virtual Input Device Placement On ATouch Screen User Interface,” filed Sep. 16, 2005; (7) U.S. patentapplication Ser. No. 11/228,700, “Operation Of A Computer With A TouchScreen Interface,” filed Sep. 16, 2005; (8) U.S. patent application Ser.No. 11/228,737, “Activating Virtual Keys Of A Touch-Screen VirtualKeyboard,” filed Sep. 16, 2005; and (9) U.S. patent application Ser. No.11/367,749, “Multi-Functional Hand-Held Device,” filed Mar. 3, 2006. Allof these applications are incorporated by reference herein in theirentirety.

Touch screen 112 optionally has a video resolution in excess of 100 dpi.In some embodiments, the touch screen has a video resolution ofapproximately 160 dpi. The user optionally makes contact with touchscreen 112 using any suitable object or appendage, such as a stylus, afinger, and so forth. In some embodiments, the user interface isdesigned to work primarily with finger-based contacts and gestures,which can be less precise than stylus-based input due to the larger areaof contact of a finger on the touch screen. In some embodiments, thedevice translates the rough finger-based input into a precisepointer/cursor position or command for performing the actions desired bythe user.

In some embodiments, in addition to the touch screen, device 100optionally includes a touchpad (not shown) for activating ordeactivating particular functions. In some embodiments, the touchpad isa touch-sensitive area of the device that, unlike the touch screen, doesnot display visual output. The touchpad is, optionally, atouch-sensitive surface that is separate from touch screen 112 or anextension of the touch-sensitive surface formed by the touch screen.

Device 100 also includes power system 162 for powering the variouscomponents. Power system 162 optionally includes a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light-emittingdiode (LED)) and any other components associated with the generation,management and distribution of power in portable devices.

Device 100 optionally also includes one or more optical sensors 164.FIG. 1A shows an optical sensor coupled to optical sensor controller 158in I/O subsystem 106. Optical sensor 164 optionally includescharge-coupled device (CCD) or complementary metal-oxide semiconductor(CMOS) phototransistors. Optical sensor 164 receives light from theenvironment, projected through one or more lenses, and converts thelight to data representing an image. In conjunction with imaging module143 (also called a camera module), optical sensor 164 optionallycaptures still images or video. In some embodiments, an optical sensoris located on the back of device 100, opposite touch screen display 112on the front of the device so that the touch screen display is enabledfor use as a viewfinder for still and/or video image acquisition. Insome embodiments, an optical sensor is located on the front of thedevice so that the user's image is, optionally, obtained for videoconferencing while the user views the other video conferenceparticipants on the touch screen display. In some embodiments, theposition of optical sensor 164 can be changed by the user (e.g., byrotating the lens and the sensor in the device housing) so that a singleoptical sensor 164 is used along with the touch screen display for bothvideo conferencing and still and/or video image acquisition.

Device 100 optionally also includes one or more contact intensitysensors 165. FIG. 1A shows a contact intensity sensor coupled tointensity sensor controller 159 in I/O subsystem 106. Contact intensitysensor 165 optionally includes one or more piezoresistive strain gauges,capacitive force sensors, electric force sensors, piezoelectric forcesensors, optical force sensors, capacitive touch-sensitive surfaces, orother intensity sensors (e.g., sensors used to measure the force (orpressure) of a contact on a touch-sensitive surface). Contact intensitysensor 165 receives contact intensity information (e.g., pressureinformation or a proxy for pressure information) from the environment.In some embodiments, at least one contact intensity sensor is collocatedwith, or proximate to, a touch-sensitive surface (e.g., touch-sensitivedisplay system 112). In some embodiments, at least one contact intensitysensor is located on the back of device 100, opposite touch screendisplay 112, which is located on the front of device 100.

Device 100 optionally also includes one or more proximity sensors 166.FIG. 1A shows proximity sensor 166 coupled to peripherals interface 118.Alternately, proximity sensor 166 is, optionally, coupled to inputcontroller 160 in I/O subsystem 106. Proximity sensor 166 optionallyperforms as described in U.S. patent application Ser. No. 11/241,839,“Proximity Detector In Handheld Device”; Ser. No. 11/240,788, “ProximityDetector In Handheld Device”; Ser. No. 11/620,702, “Using Ambient LightSensor To Augment Proximity Sensor Output”; Ser. No. 11/586,862,“Automated Response To And Sensing Of User Activity In PortableDevices”; and Ser. No. 11/638,251, “Methods And Systems For AutomaticConfiguration Of Peripherals,” which are hereby incorporated byreference in their entirety. In some embodiments, the proximity sensorturns off and disables touch screen 112 when the multifunction device isplaced near the user's ear (e.g., when the user is making a phone call).

Device 100 optionally also includes one or more tactile outputgenerators 167. FIG. 1A shows a tactile output generator coupled tohaptic feedback controller 161 in I/O subsystem 106. Tactile outputgenerator 167 optionally includes one or more electroacoustic devicessuch as speakers or other audio components and/or electromechanicaldevices that convert energy into linear motion such as a motor,solenoid, electroactive polymer, piezoelectric actuator, electrostaticactuator, or other tactile output generating component (e.g., acomponent that converts electrical signals into tactile outputs on thedevice). Contact intensity sensor 165 receives tactile feedbackgeneration instructions from haptic feedback module 133 and generatestactile outputs on device 100 that are capable of being sensed by a userof device 100. In some embodiments, at least one tactile outputgenerator is collocated with, or proximate to, a touch-sensitive surface(e.g., touch-sensitive display system 112) and, optionally, generates atactile output by moving the touch-sensitive surface vertically (e.g.,in/out of a surface of device 100) or laterally (e.g., back and forth inthe same plane as a surface of device 100). In some embodiments, atleast one tactile output generator sensor is located on the back ofdevice 100, opposite touch screen display 112, which is located on thefront of device 100.

Device 100 optionally also includes one or more accelerometers 168. FIG.1A shows accelerometer 168 coupled to peripherals interface 118.Alternately, accelerometer 168 is, optionally, coupled to an inputcontroller 160 in I/O subsystem 106. Accelerometer 168 optionallyperforms as described in U.S. Patent Publication No. 20050190059,“Acceleration-based Theft Detection System for Portable ElectronicDevices,” and U.S. Patent Publication No. 20060017692, “Methods AndApparatuses For Operating A Portable Device Based On An Accelerometer,”both of which are incorporated by reference herein in their entirety. Insome embodiments, information is displayed on the touch screen displayin a portrait view or a landscape view based on an analysis of datareceived from the one or more accelerometers. Device 100 optionallyincludes, in addition to accelerometer(s) 168, a magnetometer (notshown) and a GPS (or GLONASS or other global navigation system) receiver(not shown) for obtaining information concerning the location andorientation (e.g., portrait or landscape) of device 100.

In some embodiments, the software components stored in memory 102include operating system 126, communication module (or set ofinstructions) 128, contact/motion module (or set of instructions) 130,graphics module (or set of instructions) 132, text input module (or setof instructions) 134, Global Positioning System (GPS) module (or set ofinstructions) 135, and applications (or sets of instructions) 136.Furthermore, in some embodiments, memory 102 (FIG. 1A) or 370 (FIG. 3)stores device/global internal state 157, as shown in FIGS. 1A and 3.Device/global internal state 157 includes one or more of: activeapplication state, indicating which applications, if any, are currentlyactive; display state, indicating what applications, views or otherinformation occupy various regions of touch screen display 112; sensorstate, including information obtained from the device's various sensorsand input control devices 116; and location information concerning thedevice's location and/or attitude.

Operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X, iOS,WINDOWS, or an embedded operating system such as VxWorks) includesvarious software components and/or drivers for controlling and managinggeneral system tasks (e.g., memory management, storage device control,power management, etc.) and facilitates communication between varioushardware and software components.

Communication module 128 facilitates communication with other devicesover one or more external ports 124 and also includes various softwarecomponents for handling data received by RF circuitry 108 and/orexternal port 124. External port 124 (e.g., Universal Serial Bus (USB),FIREWIRE, etc.) is adapted for coupling directly to other devices orindirectly over a network (e.g., the Internet, wireless LAN, etc.). Insome embodiments, the external port is a multi-pin (e.g., 30-pin)connector that is the same as, or similar to and/or compatible with, the30-pin connector used on iPod® (trademark of Apple Inc.) devices.

Contact/motion module 130 optionally detects contact with touch screen112 (in conjunction with display controller 156) and othertouch-sensitive devices (e.g., a touchpad or physical click wheel).Contact/motion module 130 includes various software components forperforming various operations related to detection of contact, such asdetermining if contact has occurred (e.g., detecting a finger-downevent), determining an intensity of the contact (e.g., the force orpressure of the contact or a substitute for the force or pressure of thecontact), determining if there is movement of the contact and trackingthe movement across the touch-sensitive surface (e.g., detecting one ormore finger-dragging events), and determining if the contact has ceased(e.g., detecting a finger-up event or a break in contact).Contact/motion module 130 receives contact data from the touch-sensitivesurface. Determining movement of the point of contact, which isrepresented by a series of contact data, optionally includes determiningspeed (magnitude), velocity (magnitude and direction), and/or anacceleration (a change in magnitude and/or direction) of the point ofcontact. These operations are, optionally, applied to single contacts(e.g., one finger contacts) or to multiple simultaneous contacts (e.g.,“multitouch”/multiple finger contacts). In some embodiments,contact/motion module 130 and display controller 156 detect contact on atouchpad.

In some embodiments, contact/motion module 130 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has “clicked” onan icon). In some embodiments, at least a subset of the intensitythresholds are determined in accordance with software parameters (e.g.,the intensity thresholds are not determined by the activation thresholdsof particular physical actuators and can be adjusted without changingthe physical hardware of device 100). For example, a mouse “click”threshold of a trackpad or touch screen display can be set to any of alarge range of predefined threshold values without changing the trackpador touch screen display hardware. Additionally, in some implementations,a user of the device is provided with software settings for adjustingone or more of the set of intensity thresholds (e.g., by adjustingindividual intensity thresholds and/or by adjusting a plurality ofintensity thresholds at once with a system-level click “intensity”parameter).

Contact/motion module 130 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns (e.g., different motions, timings, and/or intensities ofdetected contacts). Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap gesture includes detecting a finger-down event followed by detectinga finger-up (liftoff) event at the same position (or substantially thesame position) as the finger-down event (e.g., at the position of anicon). As another example, detecting a finger swipe gesture on thetouch-sensitive surface includes detecting a finger-down event followedby detecting one or more finger-dragging events, and subsequentlyfollowed by detecting a finger-up (liftoff) event.

Graphics module 132 includes various known software components forrendering and displaying graphics on touch screen 112 or other display,including components for changing the visual impact (e.g., brightness,transparency, saturation, contrast, or other visual property) ofgraphics that are displayed. As used herein, the term “graphics”includes any object that can be displayed to a user, including, withoutlimitation, text, web pages, icons (such as user-interface objectsincluding soft keys), digital images, videos, animations, and the like.

In some embodiments, graphics module 132 stores data representinggraphics to be used. Each graphic is, optionally, assigned acorresponding code. Graphics module 132 receives, from applicationsetc., one or more codes specifying graphics to be displayed along with,if necessary, coordinate data and other graphic property data, and thengenerates screen image data to output to display controller 156.

Haptic feedback module 133 includes various software components forgenerating instructions used by tactile output generator(s) 167 toproduce tactile outputs at one or more locations on device 100 inresponse to user interactions with device 100.

Text input module 134, which is, optionally, a component of graphicsmodule 132, provides soft keyboards for entering text in variousapplications (e.g., contacts 137, e-mail 140, IM 141, browser 147, andany other application that needs text input).

GPS module 135 determines the location of the device and provides thisinformation for use in various applications (e.g., to telephone 138 foruse in location-based dialing; to camera 143 as picture/video metadata;and to applications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

Applications 136 optionally include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   -   Contacts module 137 (sometimes called an address book or contact        list);    -   Telephone module 138;    -   Video conference module 139;    -   E-mail client module 140;    -   Instant messaging (IM) module 141;    -   Workout support module 142;    -   Camera module 143 for still and/or video images;    -   Image management module 144;    -   Video player module;    -   Music player module;    -   Browser module 147;    -   Calendar module 148;    -   Widget modules 149, which optionally include one or more of:        weather widget 149-1, stocks widget 149-2, calculator widget        149-3, alarm clock widget 149-4, dictionary widget 149-5, and        other widgets obtained by the user, as well as user-created        widgets 149-6;    -   Widget creator module 150 for making user-created widgets 149-6;    -   Search module 151;    -   Video and music player module 152, which merges video player        module and music player module;    -   Notes module 153;    -   Map module 154; and/or    -   Online video module 155.

Examples of other applications 136 that are, optionally, stored inmemory 102 include other word processing applications, other imageediting applications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, and text input module134, contacts module 137 are, optionally, used to manage an address bookor contact list (e.g., stored in application internal state 192 ofcontacts module 137 in memory 102 or memory 370), including: addingname(s) to the address book; deleting name(s) from the address book;associating telephone number(s), e-mail address(es), physicaladdress(es) or other information with a name; associating an image witha name; categorizing and sorting names; providing telephone numbers ore-mail addresses to initiate and/or facilitate communications bytelephone 138, video conference module 139, e-mail 140, or IM 141; andso forth.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch screen 112, display controller 156, contact/motionmodule 130, graphics module 132, and text input module 134, telephonemodule 138 are optionally, used to enter a sequence of characterscorresponding to a telephone number, access one or more telephonenumbers in contacts module 137, modify a telephone number that has beenentered, dial a respective telephone number, conduct a conversation, anddisconnect or hang up when the conversation is completed. As notedabove, the wireless communication optionally uses any of a plurality ofcommunications standards, protocols, and technologies.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch screen 112, display controller 156, optical sensor164, optical sensor controller 158, contact/motion module 130, graphicsmodule 132, text input module 134, contacts module 137, and telephonemodule 138, video conference module 139 includes executable instructionsto initiate, conduct, and terminate a video conference between a userand one or more other participants in accordance with user instructions.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, and textinput module 134, e-mail client module 140 includes executableinstructions to create, send, receive, and manage e-mail in response touser instructions. In conjunction with image management module 144,e-mail client module 140 makes it very easy to create and send e-mailswith still or video images taken with camera module 143.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, and textinput module 134, the instant messaging module 141 includes executableinstructions to enter a sequence of characters corresponding to aninstant message, to modify previously entered characters, to transmit arespective instant message (for example, using a Short Message Service(SMS) or Multimedia Message Service (MMS) protocol for telephony-basedinstant messages or using XMPP, SIMPLE, or IMPS for Internet-basedinstant messages), to receive instant messages, and to view receivedinstant messages. In some embodiments, transmitted and/or receivedinstant messages optionally include graphics, photos, audio files, videofiles and/or other attachments as are supported in an MMS and/or anEnhanced Messaging Service (EMS). As used herein, “instant messaging”refers to both telephony-based messages (e.g., messages sent using SMSor MMS) and Internet-based messages (e.g., messages sent using XMPP,SIMPLE, or IMPS).

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, textinput module 134, GPS module 135, map module 154, and music playermodule, workout support module 142 includes executable instructions tocreate workouts (e.g., with time, distance, and/or calorie burninggoals); communicate with workout sensors (sports devices); receiveworkout sensor data; calibrate sensors used to monitor a workout; selectand play music for a workout; and display, store, and transmit workoutdata.

In conjunction with touch screen 112, display controller 156, opticalsensor(s) 164, optical sensor controller 158, contact/motion module 130,graphics module 132, and image management module 144, camera module 143includes executable instructions to capture still images or video(including a video stream) and store them into memory 102, modifycharacteristics of a still image or video, or delete a still image orvideo from memory 102.

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, text input module 134,and camera module 143, image management module 144 includes executableinstructions to arrange, modify (e.g., edit), or otherwise manipulate,label, delete, present (e.g., in a digital slide show or album), andstore still and/or video images.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, and textinput module 134, browser module 147 includes executable instructions tobrowse the Internet in accordance with user instructions, includingsearching, linking to, receiving, and displaying web pages or portionsthereof, as well as attachments and other files linked to web pages.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, textinput module 134, e-mail client module 140, and browser module 147,calendar module 148 includes executable instructions to create, display,modify, and store calendars and data associated with calendars (e.g.,calendar entries, to-do lists, etc.) in accordance with userinstructions.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, textinput module 134, and browser module 147, widget modules 149 aremini-applications that are, optionally, downloaded and used by a user(e.g., weather widget 149-1, stocks widget 149-2, calculator widget149-3, alarm clock widget 149-4, and dictionary widget 149-5) or createdby the user (e.g., user-created widget 149-6). In some embodiments, awidget includes an HTML (Hypertext Markup Language) file, a CSS(Cascading Style Sheets) file, and a JavaScript file. In someembodiments, a widget includes an XML (Extensible Markup Language) fileand a JavaScript file (e.g., Yahoo! Widgets).

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, textinput module 134, and browser module 147, the widget creator module 150are, optionally, used by a user to create widgets (e.g., turning auser-specified portion of a web page into a widget).

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, and text input module134, search module 151 includes executable instructions to search fortext, music, sound, image, video, and/or other files in memory 102 thatmatch one or more search criteria (e.g., one or more user-specifiedsearch terms) in accordance with user instructions.

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, audio circuitry 110,speaker 111, RF circuitry 108, and browser module 147, video and musicplayer module 152 includes executable instructions that allow the userto download and play back recorded music and other sound files stored inone or more file formats, such as MP3 or AAC files, and executableinstructions to display, present, or otherwise play back videos (e.g.,on touch screen 112 or on an external, connected display via externalport 124). In some embodiments, device 100 optionally includes thefunctionality of an MP3 player, such as an iPod (trademark of AppleInc.).

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, and text input module134, notes module 153 includes executable instructions to create andmanage notes, to-do lists, and the like in accordance with userinstructions.

In conjunction with RF circuitry 108, touch screen 112, displaycontroller 156, contact/motion module 130, graphics module 132, textinput module 134, GPS module 135, and browser module 147, map module 154are, optionally, used to receive, display, modify, and store maps anddata associated with maps (e.g., driving directions, data on stores andother points of interest at or near a particular location, and otherlocation-based data) in accordance with user instructions.

In conjunction with touch screen 112, display controller 156,contact/motion module 130, graphics module 132, audio circuitry 110,speaker 111, RF circuitry 108, text input module 134, e-mail clientmodule 140, and browser module 147, online video module 155 includesinstructions that allow the user to access, browse, receive (e.g., bystreaming and/or download), play back (e.g., on the touch screen or onan external, connected display via external port 124), send an e-mailwith a link to a particular online video, and otherwise manage onlinevideos in one or more file formats, such as H.264. In some embodiments,instant messaging module 141, rather than e-mail client module 140, isused to send a link to a particular online video. Additional descriptionof the online video application can be found in U.S. Provisional PatentApplication No. 60/936,562, “Portable Multifunction Device, Method, andGraphical User Interface for Playing Online Videos,” filed Jun. 20,2007, and U.S. patent application Ser. No. 11/968,067, “PortableMultifunction Device, Method, and Graphical User Interface for PlayingOnline Videos,” filed Dec. 31, 2007, the contents of which are herebyincorporated by reference in their entirety.

Each of the above-identified modules and applications corresponds to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (e.g., sets of instructions) need notbe implemented as separate software programs, procedures, or modules,and thus various subsets of these modules are, optionally, combined orotherwise rearranged in various embodiments. For example, video playermodule is, optionally, combined with music player module into a singlemodule (e.g., video and music player module 152, FIG. 1A). In someembodiments, memory 102 optionally stores a subset of the modules anddata structures identified above. Furthermore, memory 102 optionallystores additional modules and data structures not described above.

In some embodiments, device 100 is a device where operation of apredefined set of functions on the device is performed exclusivelythrough a touch screen and/or a touchpad. By using a touch screen and/ora touchpad as the primary input control device for operation of device100, the number of physical input control devices (such as push buttons,dials, and the like) on device 100 is, optionally, reduced.

The predefined set of functions that are performed exclusively through atouch screen and/or a touchpad optionally include navigation betweenuser interfaces. In some embodiments, the touchpad, when touched by theuser, navigates device 100 to a main, home, or root menu from any userinterface that is displayed on device 100. In such embodiments, a “menubutton” is implemented using a touchpad. In some other embodiments, themenu button is a physical push button or other physical input controldevice instead of a touchpad.

FIG. 1B is a block diagram illustrating exemplary components for eventhandling in accordance with some embodiments. In some embodiments,memory 102 (FIG. 1A) or 370 (FIG. 3) includes event sorter 170 (e.g., inoperating system 126) and a respective application 136-1 (e.g., any ofthe aforementioned applications 137-151, 155, 380-390).

Event sorter 170 receives event information and determines theapplication 136-1 and application view 191 of application 136-1 to whichto deliver the event information. Event sorter 170 includes eventmonitor 171 and event dispatcher module 174. In some embodiments,application 136-1 includes application internal state 192, whichindicates the current application view(s) displayed on touch-sensitivedisplay 112 when the application is active or executing. In someembodiments, device/global internal state 157 is used by event sorter170 to determine which application(s) is (are) currently active, andapplication internal state 192 is used by event sorter 170 to determineapplication views 191 to which to deliver event information.

In some embodiments, application internal state 192 includes additionalinformation, such as one or more of: resume information to be used whenapplication 136-1 resumes execution, user interface state informationthat indicates information being displayed or that is ready for displayby application 136-1, a state queue for enabling the user to go back toa prior state or view of application 136-1, and a redo/undo queue ofprevious actions taken by the user.

Event monitor 171 receives event information from peripherals interface118. Event information includes information about a sub-event (e.g., auser touch on touch-sensitive display 112, as part of a multi-touchgesture). Peripherals interface 118 transmits information it receivesfrom I/O subsystem 106 or a sensor, such as proximity sensor 166,accelerometer(s) 168, and/or microphone 113 (through audio circuitry110). Information that peripherals interface 118 receives from I/Osubsystem 106 includes information from touch-sensitive display 112 or atouch-sensitive surface.

In some embodiments, event monitor 171 sends requests to the peripheralsinterface 118 at predetermined intervals. In response, peripheralsinterface 118 transmits event information. In other embodiments,peripherals interface 118 transmits event information only when there isa significant event (e.g., receiving an input above a predeterminednoise threshold and/or for more than a predetermined duration).

In some embodiments, event sorter 170 also includes a hit viewdetermination module 172 and/or an active event recognizer determinationmodule 173.

Hit view determination module 172 provides software procedures fordetermining where a sub-event has taken place within one or more viewswhen touch-sensitive display 112 displays more than one view. Views aremade up of controls and other elements that a user can see on thedisplay.

Another aspect of the user interface associated with an application is aset of views, sometimes herein called application views or userinterface windows, in which information is displayed and touch-basedgestures occur. The application views (of a respective application) inwhich a touch is detected optionally correspond to programmatic levelswithin a programmatic or view hierarchy of the application. For example,the lowest level view in which a touch is detected is, optionally,called the hit view, and the set of events that are recognized as properinputs are, optionally, determined based, at least in part, on the hitview of the initial touch that begins a touch-based gesture.

Hit view determination module 172 receives information related tosub-events of a touch-based gesture. When an application has multipleviews organized in a hierarchy, hit view determination module 172identifies a hit view as the lowest view in the hierarchy which shouldhandle the sub-event. In most circumstances, the hit view is the lowestlevel view in which an initiating sub-event occurs (e.g., the firstsub-event in the sequence of sub-events that form an event or potentialevent). Once the hit view is identified by the hit view determinationmodule 172, the hit view typically receives all sub-events related tothe same touch or input source for which it was identified as the hitview.

Active event recognizer determination module 173 determines which viewor views within a view hierarchy should receive a particular sequence ofsub-events. In some embodiments, active event recognizer determinationmodule 173 determines that only the hit view should receive a particularsequence of sub-events. In other embodiments, active event recognizerdetermination module 173 determines that all views that include thephysical location of a sub-event are actively involved views, andtherefore determines that all actively involved views should receive aparticular sequence of sub-events. In other embodiments, even if touchsub-events were entirely confined to the area associated with oneparticular view, views higher in the hierarchy would still remain asactively involved views.

Event dispatcher module 174 dispatches the event information to an eventrecognizer (e.g., event recognizer 180). In embodiments including activeevent recognizer determination module 173, event dispatcher module 174delivers the event information to an event recognizer determined byactive event recognizer determination module 173. In some embodiments,event dispatcher module 174 stores in an event queue the eventinformation, which is retrieved by a respective event receiver 182.

In some embodiments, operating system 126 includes event sorter 170.Alternatively, application 136-1 includes event sorter 170. In yet otherembodiments, event sorter 170 is a stand-alone module, or a part ofanother module stored in memory 102, such as contact/motion module 130.

In some embodiments, application 136-1 includes a plurality of eventhandlers 190 and one or more application views 191, each of whichincludes instructions for handling touch events that occur within arespective view of the application's user interface. Each applicationview 191 of the application 136-1 includes one or more event recognizers180. Typically, a respective application view 191 includes a pluralityof event recognizers 180. In other embodiments, one or more of eventrecognizers 180 are part of a separate module, such as a user interfacekit (not shown) or a higher level object from which application 136-1inherits methods and other properties. In some embodiments, a respectiveevent handler 190 includes one or more of: data updater 176, objectupdater 177, GUI updater 178, and/or event data 179 received from eventsorter 170. Event handler 190 optionally utilizes or calls data updater176, object updater 177, or GUI updater 178 to update the applicationinternal state 192. Alternatively, one or more of the application views191 include one or more respective event handlers 190. Also, in someembodiments, one or more of data updater 176, object updater 177, andGUI updater 178 are included in a respective application view 191.

A respective event recognizer 180 receives event information (e.g.,event data 179) from event sorter 170 and identifies an event from theevent information. Event recognizer 180 includes event receiver 182 andevent comparator 184. In some embodiments, event recognizer 180 alsoincludes at least a subset of: metadata 183, and event deliveryinstructions 188 (which optionally include sub-event deliveryinstructions).

Event receiver 182 receives event information from event sorter 170. Theevent information includes information about a sub-event, for example, atouch or a touch movement. Depending on the sub-event, the eventinformation also includes additional information, such as location ofthe sub-event. When the sub-event concerns motion of a touch, the eventinformation optionally also includes speed and direction of thesub-event. In some embodiments, events include rotation of the devicefrom one orientation to another (e.g., from a portrait orientation to alandscape orientation, or vice versa), and the event informationincludes corresponding information about the current orientation (alsocalled device attitude) of the device.

Event comparator 184 compares the event information to predefined eventor sub-event definitions and, based on the comparison, determines anevent or sub-event, or determines or updates the state of an event orsub-event. In some embodiments, event comparator 184 includes eventdefinitions 186. Event definitions 186 contain definitions of events(e.g., predefined sequences of sub-events), for example, event 1(187-1), event 2 (187-2), and others. In some embodiments, sub-events inan event (187) include, for example, touch begin, touch end, touchmovement, touch cancellation, and multiple touching. In one example, thedefinition for event 1 (187-1) is a double tap on a displayed object.The double tap, for example, comprises a first touch (touch begin) onthe displayed object for a predetermined phase, a first liftoff (touchend) for a predetermined phase, a second touch (touch begin) on thedisplayed object for a predetermined phase, and a second liftoff (touchend) for a predetermined phase. In another example, the definition forevent 2 (187-2) is a dragging on a displayed object. The dragging, forexample, comprises a touch (or contact) on the displayed object for apredetermined phase, a movement of the touch across touch-sensitivedisplay 112, and liftoff of the touch (touch end). In some embodiments,the event also includes information for one or more associated eventhandlers 190.

In some embodiments, event definition 187 includes a definition of anevent for a respective user-interface object. In some embodiments, eventcomparator 184 performs a hit test to determine which user-interfaceobject is associated with a sub-event. For example, in an applicationview in which three user-interface objects are displayed ontouch-sensitive display 112, when a touch is detected on touch-sensitivedisplay 112, event comparator 184 performs a hit test to determine whichof the three user-interface objects is associated with the touch(sub-event). If each displayed object is associated with a respectiveevent handler 190, the event comparator uses the result of the hit testto determine which event handler 190 should be activated. For example,event comparator 184 selects an event handler associated with thesub-event and the object triggering the hit test.

In some embodiments, the definition for a respective event (187) alsoincludes delayed actions that delay delivery of the event informationuntil after it has been determined whether the sequence of sub-eventsdoes or does not correspond to the event recognizer's event type.

When a respective event recognizer 180 determines that the series ofsub-events do not match any of the events in event definitions 186, therespective event recognizer 180 enters an event impossible, eventfailed, or event ended state, after which it disregards subsequentsub-events of the touch-based gesture. In this situation, other eventrecognizers, if any, that remain active for the hit view continue totrack and process sub-events of an ongoing touch-based gesture.

In some embodiments, a respective event recognizer 180 includes metadata183 with configurable properties, flags, and/or lists that indicate howthe event delivery system should perform sub-event delivery to activelyinvolved event recognizers. In some embodiments, metadata 183 includesconfigurable properties, flags, and/or lists that indicate how eventrecognizers interact, or are enabled to interact, with one another. Insome embodiments, metadata 183 includes configurable properties, flags,and/or lists that indicate whether sub-events are delivered to varyinglevels in the view or programmatic hierarchy.

In some embodiments, a respective event recognizer 180 activates eventhandler 190 associated with an event when one or more particularsub-events of an event are recognized. In some embodiments, a respectiveevent recognizer 180 delivers event information associated with theevent to event handler 190. Activating an event handler 190 is distinctfrom sending (and deferred sending) sub-events to a respective hit view.In some embodiments, event recognizer 180 throws a flag associated withthe recognized event, and event handler 190 associated with the flagcatches the flag and performs a predefined process.

In some embodiments, event delivery instructions 188 include sub-eventdelivery instructions that deliver event information about a sub-eventwithout activating an event handler. Instead, the sub-event deliveryinstructions deliver event information to event handlers associated withthe series of sub-events or to actively involved views. Event handlersassociated with the series of sub-events or with actively involved viewsreceive the event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used inapplication 136-1. For example, data updater 176 updates the telephonenumber used in contacts module 137, or stores a video file used in videoplayer module. In some embodiments, object updater 177 creates andupdates objects used in application 136-1. For example, object updater177 creates a new user-interface object or updates the position of auser-interface object. GUI updater 178 updates the GUI. For example, GUIupdater 178 prepares display information and sends it to graphics module132 for display on a touch-sensitive display.

In some embodiments, event handler(s) 190 includes or has access to dataupdater 176, object updater 177, and GUI updater 178. In someembodiments, data updater 176, object updater 177, and GUI updater 178are included in a single module of a respective application 136-1 orapplication view 191. In other embodiments, they are included in two ormore software modules.

It shall be understood that the foregoing discussion regarding eventhandling of user touches on touch-sensitive displays also applies toother forms of user inputs to operate multifunction devices 100 withinput devices, not all of which are initiated on touch screens. Forexample, mouse movement and mouse button presses, optionally coordinatedwith single or multiple keyboard presses or holds; contact movementssuch as taps, drags, scrolls, etc. on touchpads; pen stylus inputs;movement of the device; oral instructions; detected eye movements;biometric inputs; and/or any combination thereof are optionally utilizedas inputs corresponding to sub-events which define an event to berecognized.

FIG. 2 illustrates a portable multifunction device 100 having a touchscreen 112 in accordance with some embodiments. The touch screenoptionally displays one or more graphics within user interface (UI) 200.In this embodiment, as well as others described below, a user is enabledto select one or more of the graphics by making a gesture on thegraphics, for example, with one or more fingers 202 (not drawn to scalein the figure) or one or more styluses 203 (not drawn to scale in thefigure). In some embodiments, selection of one or more graphics occurswhen the user breaks contact with the one or more graphics. In someembodiments, the gesture optionally includes one or more taps, one ormore swipes (from left to right, right to left, upward and/or downward),and/or a rolling of a finger (from right to left, left to right, upwardand/or downward) that has made contact with device 100. In someimplementations or circumstances, inadvertent contact with a graphicdoes not select the graphic. For example, a swipe gesture that sweepsover an application icon optionally does not select the correspondingapplication when the gesture corresponding to selection is a tap.

Device 100 optionally also include one or more physical buttons, such as“home” or menu button 204. As described previously, menu button 204 is,optionally, used to navigate to any application 136 in a set ofapplications that are, optionally, executed on device 100.Alternatively, in some embodiments, the menu button is implemented as asoft key in a GUI displayed on touch screen 112.

In some embodiments, device 100 includes touch screen 112, menu button204, push button 206 for powering the device on/off and locking thedevice, volume adjustment button(s) 208, subscriber identity module(SIM) card slot 210, headset jack 212, and docking/charging externalport 124. Push button 206 is, optionally, used to turn the power on/offon the device by depressing the button and holding the button in thedepressed state for a predefined time interval; to lock the device bydepressing the button and releasing the button before the predefinedtime interval has elapsed; and/or to unlock the device or initiate anunlock process. In an alternative embodiment, device 100 also acceptsverbal input for activation or deactivation of some functions throughmicrophone 113. Device 100 also, optionally, includes one or morecontact intensity sensors 165 for detecting intensity of contacts ontouch screen 112 and/or one or more tactile output generators 167 forgenerating tactile outputs for a user of device 100.

FIG. 3 is a block diagram of an exemplary multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments. Device 300 need not be portable. In some embodiments,device 300 is a laptop computer, a desktop computer, a tablet computer,a multimedia player device, a navigation device, an educational device(such as a child's learning toy), a gaming system, or a control device(e.g., a home or industrial controller). Device 300 typically includesone or more processing units (CPUs) 310, one or more network or othercommunications interfaces 360, memory 370, and one or more communicationbuses 320 for interconnecting these components. Communication buses 320optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components.Device 300 includes input/output (I/O) interface 330 comprising display340, which is typically a touch screen display. I/O interface 330 alsooptionally includes a keyboard and/or mouse (or other pointing device)350 and touchpad 355, tactile output generator 357 for generatingtactile outputs on device 300 (e.g., similar to tactile outputgenerator(s) 167 described above with reference to FIG. 1A), sensors 359(e.g., optical, acceleration, proximity, touch-sensitive, and/or contactintensity sensors similar to contact intensity sensor(s) 165 describedabove with reference to FIG. 1A). Memory 370 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM, or other random access solidstate memory devices; and optionally includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid state storagedevices. Memory 370 optionally includes one or more storage devicesremotely located from CPU(s) 310. In some embodiments, memory 370 storesprograms, modules, and data structures analogous to the programs,modules, and data structures stored in memory 102 of portablemultifunction device 100 (FIG. 1A), or a subset thereof. Furthermore,memory 370 optionally stores additional programs, modules, and datastructures not present in memory 102 of portable multifunction device100. For example, memory 370 of device 300 optionally stores drawingmodule 380, presentation module 382, word processing module 384, websitecreation module 386, disk authoring module 388, and/or spreadsheetmodule 390, while memory 102 of portable multifunction device 100 (FIG.1A) optionally does not store these modules.

Each of the above-identified elements in FIG. 3 is, optionally, storedin one or more of the previously mentioned memory devices. Each of theabove-identified modules corresponds to a set of instructions forperforming a function described above. The above-identified modules orprograms (e.g., sets of instructions) need not be implemented asseparate software programs, procedures, or modules, and thus varioussubsets of these modules are, optionally, combined or otherwiserearranged in various embodiments. In some embodiments, memory 370optionally stores a subset of the modules and data structures identifiedabove. Furthermore, memory 370 optionally stores additional modules anddata structures not described above.

Attention is now directed towards embodiments of user interfaces thatare, optionally, implemented on, for example, portable multifunctiondevice 100.

FIG. 4A illustrates an exemplary user interface for a menu ofapplications on portable multifunction device 100 in accordance withsome embodiments. Similar user interfaces are, optionally, implementedon device 300. In some embodiments, user interface 400 includes thefollowing elements, or a subset or superset thereof:

-   -   Signal strength indicator(s) 402 for wireless communication(s),        such as cellular and WiFi signals;    -   Time 404;    -   Bluetooth indicator 405;    -   Battery status indicator 406;    -   Tray 408 with icons for frequently used applications, such as:        -   Icon 416 for telephone module 138, labeled “Phone,” which            optionally includes an indicator 414 of the number of missed            calls or voicemail messages;        -   Icon 418 for e-mail client module 140, labeled “Mail,” which            optionally includes an indicator 410 of the number of unread            e-mails;        -   Icon 420 for browser module 147, labeled “Browser;” and        -   Icon 422 for video and music player module 152, also            referred to as iPod (trademark of Apple Inc.) module 152,            labeled “iPod;” and    -   Icons for other applications, such as:        -   Icon 424 for IM module 141, labeled “Messages;”        -   Icon 426 for calendar module 148, labeled “Calendar;”        -   Icon 428 for image management module 144, labeled “Photos;”        -   Icon 430 for camera module 143, labeled “Camera;”        -   Icon 432 for online video module 155, labeled “Online            Video;”        -   Icon 434 for stocks widget 149-2, labeled “Stocks;”        -   Icon 436 for map module 154, labeled “Maps;”        -   Icon 438 for weather widget 149-1, labeled “Weather;”        -   Icon 440 for alarm clock widget 149-4, labeled “Clock;”        -   Icon 442 for workout support module 142, labeled “Workout            Support;”        -   Icon 444 for notes module 153, labeled “Notes;” and        -   Icon 446 for a settings application or module, labeled            “Settings,” which provides access to settings for device 100            and its various applications 136.

It should be noted that the icon labels illustrated in FIG. 4A aremerely exemplary. For example, icon 422 for video and music playermodule 152 is labeled “Music” or “Music Player.” Other labels are,optionally, used for various application icons. In some embodiments, alabel for a respective application icon includes a name of anapplication corresponding to the respective application icon. In someembodiments, a label for a particular application icon is distinct froma name of an application corresponding to the particular applicationicon.

FIG. 4B illustrates an exemplary user interface on a device (e.g.,device 300, FIG. 3) with a touch-sensitive surface 451 (e.g., a tabletor touchpad 355, FIG. 3) that is separate from the display 450 (e.g.,touch screen display 112). Device 300 also, optionally, includes one ormore contact intensity sensors (e.g., one or more of sensors 359) fordetecting intensity of contacts on touch-sensitive surface 451 and/orone or more tactile output generators 357 for generating tactile outputsfor a user of device 300.

Although some of the examples that follow will be given with referenceto inputs on touch screen display 112 (where the touch-sensitive surfaceand the display are combined), in some embodiments, the device detectsinputs on a touch-sensitive surface that is separate from the display,as shown in FIG. 4B. In some embodiments, the touch-sensitive surface(e.g., 451 in FIG. 4B) has a primary axis (e.g., 452 in FIG. 4B) thatcorresponds to a primary axis (e.g., 453 in FIG. 4B) on the display(e.g., 450). In accordance with these embodiments, the device detectscontacts (e.g., 460 and 462 in FIG. 4B) with the touch-sensitive surface451 at locations that correspond to respective locations on the display(e.g., in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470).In this way, user inputs (e.g., contacts 460 and 462, and movementsthereof) detected by the device on the touch-sensitive surface (e.g.,451 in FIG. 4B) are used by the device to manipulate the user interfaceon the display (e.g., 450 in FIG. 4B) of the multifunction device whenthe touch-sensitive surface is separate from the display. It should beunderstood that similar methods are, optionally, used for other userinterfaces described herein.

Additionally, while the following examples are given primarily withreference to finger inputs (e.g., finger contacts, finger tap gestures,finger swipe gestures), it should be understood that, in someembodiments, one or more of the finger inputs are replaced with inputfrom another input device (e.g., a mouse-based input or stylus input).For example, a swipe gesture is, optionally, replaced with a mouse click(e.g., instead of a contact) followed by movement of the cursor alongthe path of the swipe (e.g., instead of movement of the contact). Asanother example, a tap gesture is, optionally, replaced with a mouseclick while the cursor is located over the location of the tap gesture(e.g., instead of detection of the contact followed by ceasing to detectthe contact). Similarly, when multiple user inputs are simultaneouslydetected, it should be understood that multiple computer mice are,optionally, used simultaneously, or a mouse and finger contacts are,optionally, used simultaneously.

FIG. 5A illustrates exemplary personal electronic device 500. Device 500includes body 502. In some embodiments, device 500 can include some orall of the features described with respect to devices 100 and 300 (e.g.,FIGS. 1A-4B). In some embodiments, device 500 has touch-sensitivedisplay screen 504, hereafter touch screen 504. Alternatively, or inaddition to touch screen 504, device 500 has a display and atouch-sensitive surface. As with devices 100 and 300, in someembodiments, touch screen 504 (or the touch-sensitive surface)optionally includes one or more intensity sensors for detectingintensity of contacts (e.g., touches) being applied. The one or moreintensity sensors of touch screen 504 (or the touch-sensitive surface)can provide output data that represents the intensity of touches. Theuser interface of device 500 can respond to touches based on theirintensity, meaning that touches of different intensities can invokedifferent user interface operations on device 500.

Exemplary techniques for detecting and processing touch intensity arefound, for example, in related applications: International PatentApplication Serial No. PCT/US2013/040061, titled “Device, Method, andGraphical User Interface for Displaying User Interface ObjectsCorresponding to an Application,” filed May 8, 2013, published as WIPOPublication No. WO/2013/169849, and International Patent ApplicationSerial No. PCT/US2013/069483, titled “Device, Method, and Graphical UserInterface for Transitioning Between Touch Input to Display OutputRelationships,” filed Nov. 11, 2013, published as WIPO Publication No.WO/2014/105276, each of which is hereby incorporated by reference intheir entirety.

In some embodiments, device 500 has one or more input mechanisms 506 and508. Input mechanisms 506 and 508, if included, can be physical.Examples of physical input mechanisms include push buttons and rotatablemechanisms. In some embodiments, device 500 has one or more attachmentmechanisms. Such attachment mechanisms, if included, can permitattachment of device 500 with, for example, hats, eyewear, earrings,necklaces, shirts, jackets, bracelets, watch straps, chains, trousers,belts, shoes, purses, backpacks, and so forth. These attachmentmechanisms permit device 500 to be worn by a user.

FIG. 5B depicts exemplary personal electronic device 500. In someembodiments, device 500 can include some or all of the componentsdescribed with respect to FIGS. 1A, 1B, and 3. Device 500 has bus 512that operatively couples I/O section 514 with one or more computerprocessors 516 and memory 518. I/O section 514 can be connected todisplay 504, which can have touch-sensitive component 522 and,optionally, intensity sensor 524 (e.g., contact intensity sensor). Inaddition, I/O section 514 can be connected with communication unit 530for receiving application and operating system data, using Wi-Fi,Bluetooth, near field communication (NFC), cellular, and/or otherwireless communication techniques. Device 500 can include inputmechanisms 506 and/or 508. Input mechanism 506 is, optionally, arotatable input device or a depressible and rotatable input device, forexample. Input mechanism 508 is, optionally, a button, in some examples.

Input mechanism 508 is, optionally, a microphone, in some examples.Personal electronic device 500 optionally includes various sensors, suchas GPS sensor 532, accelerometer 534, directional sensor 540 (e.g.,compass), gyroscope 536, motion sensor 538, and/or a combinationthereof, all of which can be operatively connected to I/O section 514.

Memory 518 of personal electronic device 500 can include one or morenon-transitory computer-readable storage mediums, for storingcomputer-executable instructions, which, when executed by one or morecomputer processors 516, for example, can cause the computer processorsto perform the techniques described below, including processes 700,1000, 1300, 1600, 1900, 2100, 2300, 2600, 2900, and 3200 (FIGS. 7A-7C,10, 13, 16, 19, 21, 23, 26, 29, and 32). A computer-readable storagemedium can be any medium that can tangibly contain or storecomputer-executable instructions for use by or in connection with theinstruction execution system, apparatus, or device. In some examples,the storage medium is a transitory computer-readable storage medium. Insome examples, the storage medium is a non-transitory computer-readablestorage medium. The non-transitory computer-readable storage medium caninclude, but is not limited to, magnetic, optical, and/or semiconductorstorages. Examples of such storage include magnetic disks, optical discsbased on CD, DVD, or Blu-ray technologies, as well as persistentsolid-state memory such as flash, solid-state drives, and the like.Personal electronic device 500 is not limited to the components andconfiguration of FIG. 5B, but can include other or additional componentsin multiple configurations.

As used here, the term “affordance” refers to a user-interactivegraphical user interface object that is, optionally, displayed on thedisplay screen of devices 100, 300, and/or 500 (FIGS. 1A-1B, 3, and5A-5H). For example, an image (e.g., icon), a button, and text (e.g.,hyperlink) each optionally constitute an affordance.

As used herein, the term “focus selector” refers to an input elementthat indicates a current part of a user interface with which a user isinteracting. In some implementations that include a cursor or otherlocation marker, the cursor acts as a “focus selector” so that when aninput (e.g., a press input) is detected on a touch-sensitive surface(e.g., touchpad 355 in FIG. 3 or touch-sensitive surface 451 in FIG. 4B)while the cursor is over a particular user interface element (e.g., abutton, window, slider, or other user interface element), the particularuser interface element is adjusted in accordance with the detectedinput. In some implementations that include a touch screen display(e.g., touch-sensitive display system 112 in FIG. 1A or touch screen 112in FIG. 4A) that enables direct interaction with user interface elementson the touch screen display, a detected contact on the touch screen actsas a “focus selector” so that when an input (e.g., a press input by thecontact) is detected on the touch screen display at a location of aparticular user interface element (e.g., a button, window, slider, orother user interface element), the particular user interface element isadjusted in accordance with the detected input. In some implementations,focus is moved from one region of a user interface to another region ofthe user interface without corresponding movement of a cursor ormovement of a contact on a touch screen display (e.g., by using a tabkey or arrow keys to move focus from one button to another button); inthese implementations, the focus selector moves in accordance withmovement of focus between different regions of the user interface.Without regard to the specific form taken by the focus selector, thefocus selector is generally the user interface element (or contact on atouch screen display) that is controlled by the user so as tocommunicate the user's intended interaction with the user interface(e.g., by indicating, to the device, the element of the user interfacewith which the user is intending to interact). For example, the locationof a focus selector (e.g., a cursor, a contact, or a selection box) overa respective button while a press input is detected on thetouch-sensitive surface (e.g., a touchpad or touch screen) will indicatethat the user is intending to activate the respective button (as opposedto other user interface elements shown on a display of the device).

As used in the specification and claims, the term “characteristicintensity” of a contact refers to a characteristic of the contact basedon one or more intensities of the contact. In some embodiments, thecharacteristic intensity is based on multiple intensity samples. Thecharacteristic intensity is, optionally, based on a predefined number ofintensity samples, or a set of intensity samples collected during apredetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10seconds) relative to a predefined event (e.g., after detecting thecontact, prior to detecting liftoff of the contact, before or afterdetecting a start of movement of the contact, prior to detecting an endof the contact, before or after detecting an increase in intensity ofthe contact, and/or before or after detecting a decrease in intensity ofthe contact). A characteristic intensity of a contact is, optionally,based on one or more of: a maximum value of the intensities of thecontact, a mean value of the intensities of the contact, an averagevalue of the intensities of the contact, a top 10 percentile value ofthe intensities of the contact, a value at the half maximum of theintensities of the contact, a value at the 90 percent maximum of theintensities of the contact, or the like. In some embodiments, theduration of the contact is used in determining the characteristicintensity (e.g., when the characteristic intensity is an average of theintensity of the contact over time). In some embodiments, thecharacteristic intensity is compared to a set of one or more intensitythresholds to determine whether an operation has been performed by auser. For example, the set of one or more intensity thresholdsoptionally includes a first intensity threshold and a second intensitythreshold. In this example, a contact with a characteristic intensitythat does not exceed the first threshold results in a first operation, acontact with a characteristic intensity that exceeds the first intensitythreshold and does not exceed the second intensity threshold results ina second operation, and a contact with a characteristic intensity thatexceeds the second threshold results in a third operation. In someembodiments, a comparison between the characteristic intensity and oneor more thresholds is used to determine whether or not to perform one ormore operations (e.g., whether to perform a respective operation orforgo performing the respective operation), rather than being used todetermine whether to perform a first operation or a second operation.

FIG. 5C illustrates detecting a plurality of contacts 552A-552E ontouch-sensitive display screen 504 with a plurality of intensity sensors524A-524D. FIG. 5C additionally includes intensity diagrams that showthe current intensity measurements of the intensity sensors 524A-524Drelative to units of intensity. In this example, the intensitymeasurements of intensity sensors 524A and 524D are each 9 units ofintensity, and the intensity measurements of intensity sensors 524B and524C are each 7 units of intensity. In some implementations, anaggregate intensity is the sum of the intensity measurements of theplurality of intensity sensors 524A-524D, which in this example is 32intensity units. In some embodiments, each contact is assigned arespective intensity that is a portion of the aggregate intensity. FIG.5D illustrates assigning the aggregate intensity to contacts 552A-552Ebased on their distance from the center of force 554. In this example,each of contacts 552A, 552B, and 552E are assigned an intensity ofcontact of 8 intensity units of the aggregate intensity, and each ofcontacts 552C and 552D are assigned an intensity of contact of 4intensity units of the aggregate intensity. More generally, in someimplementations, each contact j is assigned a respective intensity Ijthat is a portion of the aggregate intensity, A, in accordance with apredefined mathematical function, Ij=A·(Dj/ΣDi), where Dj is thedistance of the respective contact j to the center of force, and ΣDi isthe sum of the distances of all the respective contacts (e.g., i=1 tolast) to the center of force. The operations described with reference toFIGS. 5C-5D can be performed using an electronic device similar oridentical to device 100, 300, or 500. In some embodiments, acharacteristic intensity of a contact is based on one or moreintensities of the contact. In some embodiments, the intensity sensorsare used to determine a single characteristic intensity (e.g., a singlecharacteristic intensity of a single contact). It should be noted thatthe intensity diagrams are not part of a displayed user interface, butare included in FIGS. 5C-5D to aid the reader.

In some embodiments, a portion of a gesture is identified for purposesof determining a characteristic intensity. For example, atouch-sensitive surface optionally receives a continuous swipe contacttransitioning from a start location and reaching an end location, atwhich point the intensity of the contact increases. In this example, thecharacteristic intensity of the contact at the end location is,optionally, based on only a portion of the continuous swipe contact, andnot the entire swipe contact (e.g., only the portion of the swipecontact at the end location). In some embodiments, a smoothing algorithmis, optionally, applied to the intensities of the swipe contact prior todetermining the characteristic intensity of the contact. For example,the smoothing algorithm optionally includes one or more of: anunweighted sliding-average smoothing algorithm, a triangular smoothingalgorithm, a median filter smoothing algorithm, and/or an exponentialsmoothing algorithm. In some circumstances, these smoothing algorithmseliminate narrow spikes or dips in the intensities of the swipe contactfor purposes of determining a characteristic intensity.

The intensity of a contact on the touch-sensitive surface is,optionally, characterized relative to one or more intensity thresholds,such as a contact-detection intensity threshold, a light press intensitythreshold, a deep press intensity threshold, and/or one or more otherintensity thresholds. In some embodiments, the light press intensitythreshold corresponds to an intensity at which the device will performoperations typically associated with clicking a button of a physicalmouse or a trackpad. In some embodiments, the deep press intensitythreshold corresponds to an intensity at which the device will performoperations that are different from operations typically associated withclicking a button of a physical mouse or a trackpad. In someembodiments, when a contact is detected with a characteristic intensitybelow the light press intensity threshold (e.g., and above a nominalcontact-detection intensity threshold below which the contact is nolonger detected), the device will move a focus selector in accordancewith movement of the contact on the touch-sensitive surface withoutperforming an operation associated with the light press intensitythreshold or the deep press intensity threshold. Generally, unlessotherwise stated, these intensity thresholds are consistent betweendifferent sets of user interface figures.

An increase of characteristic intensity of the contact from an intensitybelow the light press intensity threshold to an intensity between thelight press intensity threshold and the deep press intensity thresholdis sometimes referred to as a “light press” input. An increase ofcharacteristic intensity of the contact from an intensity below the deeppress intensity threshold to an intensity above the deep press intensitythreshold is sometimes referred to as a “deep press” input. An increaseof characteristic intensity of the contact from an intensity below thecontact-detection intensity threshold to an intensity between thecontact-detection intensity threshold and the light press intensitythreshold is sometimes referred to as detecting the contact on thetouch-surface. A decrease of characteristic intensity of the contactfrom an intensity above the contact-detection intensity threshold to anintensity below the contact-detection intensity threshold is sometimesreferred to as detecting liftoff of the contact from the touch-surface.In some embodiments, the contact-detection intensity threshold is zero.In some embodiments, the contact-detection intensity threshold isgreater than zero.

In some embodiments described herein, one or more operations areperformed in response to detecting a gesture that includes a respectivepress input or in response to detecting the respective press inputperformed with a respective contact (or a plurality of contacts), wherethe respective press input is detected based at least in part ondetecting an increase in intensity of the contact (or plurality ofcontacts) above a press-input intensity threshold. In some embodiments,the respective operation is performed in response to detecting theincrease in intensity of the respective contact above the press-inputintensity threshold (e.g., a “down stroke” of the respective pressinput). In some embodiments, the press input includes an increase inintensity of the respective contact above the press-input intensitythreshold and a subsequent decrease in intensity of the contact belowthe press-input intensity threshold, and the respective operation isperformed in response to detecting the subsequent decrease in intensityof the respective contact below the press-input threshold (e.g., an “upstroke” of the respective press input).

FIGS. 5E-5H illustrate detection of a gesture that includes a pressinput that corresponds to an increase in intensity of a contact 562 froman intensity below a light press intensity threshold (e.g., “IT_(L)”) inFIG. 5E, to an intensity above a deep press intensity threshold (e.g.,“IT_(D)”) in FIG. 5H. The gesture performed with contact 562 is detectedon touch-sensitive surface 560 while cursor 576 is displayed overapplication icon 572B corresponding to App 2, on a displayed userinterface 570 that includes application icons 572A-572D displayed inpredefined region 574. In some embodiments, the gesture is detected ontouch-sensitive display 504. The intensity sensors detect the intensityof contacts on touch-sensitive surface 560. The device determines thatthe intensity of contact 562 peaked above the deep press intensitythreshold (e.g., “IT_(D)”). Contact 562 is maintained on touch-sensitivesurface 560. In response to the detection of the gesture, and inaccordance with contact 562 having an intensity that goes above the deeppress intensity threshold (e.g., “IT_(D)”) during the gesture,reduced-scale representations 578A-578C (e.g., thumbnails) of recentlyopened documents for App 2 are displayed, as shown in FIGS. 5F-5H. Insome embodiments, the intensity, which is compared to the one or moreintensity thresholds, is the characteristic intensity of a contact. Itshould be noted that the intensity diagram for contact 562 is not partof a displayed user interface, but is included in FIGS. 5E-5H to aid thereader.

In some embodiments, the display of representations 578A-578C includesan animation. For example, representation 578A is initially displayed inproximity of application icon 572B, as shown in FIG. 5F. As theanimation proceeds, representation 578A moves upward and representation578B is displayed in proximity of application icon 572B, as shown inFIG. 5G. Then, representations 578A moves upward, 578B moves upwardtoward representation 578A, and representation 578C is displayed inproximity of application icon 572B, as shown in FIG. 5H. Representations578A-578C form an array above icon 572B. In some embodiments, theanimation progresses in accordance with an intensity of contact 562, asshown in FIGS. 5F-5G, where the representations 578A-578C appear andmove upwards as the intensity of contact 562 increases toward the deeppress intensity threshold (e.g., “IT_(D)”). In some embodiments, theintensity, on which the progress of the animation is based, is thecharacteristic intensity of the contact. The operations described withreference to FIGS. 5E-5H can be performed using an electronic devicesimilar or identical to device 100, 300, or 500.

In some embodiments, the device employs intensity hysteresis to avoidaccidental inputs sometimes termed “jitter,” where the device defines orselects a hysteresis intensity threshold with a predefined relationshipto the press-input intensity threshold (e.g., the hysteresis intensitythreshold is X intensity units lower than the press-input intensitythreshold or the hysteresis intensity threshold is 75%, 90%, or somereasonable proportion of the press-input intensity threshold). Thus, insome embodiments, the press input includes an increase in intensity ofthe respective contact above the press-input intensity threshold and asubsequent decrease in intensity of the contact below the hysteresisintensity threshold that corresponds to the press-input intensitythreshold, and the respective operation is performed in response todetecting the subsequent decrease in intensity of the respective contactbelow the hysteresis intensity threshold (e.g., an “up stroke” of therespective press input). Similarly, in some embodiments, the press inputis detected only when the device detects an increase in intensity of thecontact from an intensity at or below the hysteresis intensity thresholdto an intensity at or above the press-input intensity threshold and,optionally, a subsequent decrease in intensity of the contact to anintensity at or below the hysteresis intensity, and the respectiveoperation is performed in response to detecting the press input (e.g.,the increase in intensity of the contact or the decrease in intensity ofthe contact, depending on the circumstances).

For ease of explanation, the descriptions of operations performed inresponse to a press input associated with a press-input intensitythreshold or in response to a gesture including the press input are,optionally, triggered in response to detecting either: an increase inintensity of a contact above the press-input intensity threshold, anincrease in intensity of a contact from an intensity below thehysteresis intensity threshold to an intensity above the press-inputintensity threshold, a decrease in intensity of the contact below thepress-input intensity threshold, and/or a decrease in intensity of thecontact below the hysteresis intensity threshold corresponding to thepress-input intensity threshold. Additionally, in examples where anoperation is described as being performed in response to detecting adecrease in intensity of a contact below the press-input intensitythreshold, the operation is, optionally, performed in response todetecting a decrease in intensity of the contact below a hysteresisintensity threshold corresponding to, and lower than, the press-inputintensity threshold.

Attention is now directed towards embodiments of user interfaces (“UI”)and associated processes that are implemented on an electronic device,such as portable multifunction device 100, device 300, or device 500.

FIGS. 6A-6S illustrate exemplary user interfaces for managingcontrollable external devices, in accordance with some embodiments. Theuser interfaces in these figures are used to illustrate the processesdescribed below, including the processes in FIGS. 7A-7C and 23.

FIG. 6A illustrates an electronic device 600 with touch-sensitivedisplay 602. In some embodiments, device 600 includes some or all of thefeatures of devices 100, 300, and 500, discussed above. Device 600 isconfigured to receive data associated with controllable externaldevices. Device 600 is also configured to send instructions to controlcontrollable external devices.

In FIG. 6A, device 600 displays a user interface screen 610-1 formanaging controllable external devices associated with a particularlocation. In some embodiments, user interface screen 610 is included inan application for managing controllable external devices associatedwith one or more locations. Examples of locations include, but are notlimited to, an address, place, building, business, environment (e.g.,backyard, automobile, porch), and so forth. In some embodiments, thelocation includes a plurality of designated areas within or associatedwith the location, such as rooms within a location. In some embodiments,a location includes an environment model with a hierarchicalrepresentation of a physical environment (e.g., a home) that can includea lowest level of objects (e.g., rooms), as described in paragraph[0017] of filed U.S. patent application Ser. No. 14/725,912 titled“Accessory management Systems Using Environment Model,” filed May 29,2015, which is incorporated by reference above.

Device 600 is configured to access a location profile for the locationindicated on user interface screen 610-1. In some embodiments, thelocation profile is accessed from local memory on device 600 or from aremote server. A location profile includes information associated with alocation, which may include, e.g., a name, address, people associatedwith the location, an image associated with the location (e.g., awallpaper image), a year (e.g., when the location was built), notes,accessories associated with the location, current states of accessoriesat the location, automation profiles associated with the location, andscene profiles associated with the location. The terms accessory,automation profile, and scene profile are described in greater detailbelow.

In some embodiments, a user selects the location profile with user input(e.g., by selecting the location from a list of locations). In someembodiments, device 600 determines its position and determines alocation profile based on the position of device 600. In this way,device 600 automatically presents the user with information and controlsfor the location corresponding to the user's current position.

User interface screen 610-1 is a Home screen corresponding to tab 604-1and may be referred to as “Home screen 610-1”. Home screen 610-1includes an indication 606 of the location (“Ellsworth St.”). In theillustrated embodiment, indication 606 includes text. In someembodiments, an indication of a location includes an image or otheridentifying information.

Home screen 610-1 also includes a representation 620-1 of a controllableexternal device at or associated with the location. A controllableexternal device is also called an “accessory” for purposes of thisdisclosure and refers to any item or device that is configured to bemonitored and/or controlled by another device. In some embodiments, anaccessory is configured to operate in two or more states and is remotelycontrollable between the two or more states. Non-limiting examples ofaccessories include light fixtures, thermostats, outlets, fans,appliances (e.g., TV, washer, dryer, dishwasher, coffee machine), doors(e.g., door locks), automatic door openers (e.g., garage door opener),air conditioners, furnaces, still or video cameras, smoke alarms, COalarms, security systems and alarms, and so on. An accessory can includeelectronic devices that can be controlled with a special-purposecomputing device or a general-purpose computing device, such as adesktop computer, laptop computer, smart phone, other mobile phone,other handheld or wearable computing device, by providing thegeneral-purpose computing device with appropriate executable programcode or instructions, as described in paragraph [0008] of filed U.S.patent application Ser. No. 14/725,912 titled “Accessory managementSystems Using Environment Model,” filed May 29, 2015, which isincorporated by reference above.

Representation 620-1 corresponds to a thermostat accessory and includesan indication of the current state of the thermostat. In particular,representation 620-1 indicates that the current temperature is 70degrees and that the thermostat is heating to a set temperature of 72degrees. In some embodiments, the representation of an accessoryincludes an indication of an action that selection of the representationwould cause. For example, representation 620-2, which corresponds to afan accessory, includes the text “TURN ON”. This indicates thatselecting the representation will turn the fan on, and also implies thatthe fan is currently off.

Representation 620-1 is located on Home screen 610-1 under the heading“Favorite Accessories”. In some embodiments, a user can manuallydesignate an accessory as a favorite accessory. In some embodiments, anaccessory is represented as a favorite accessory based on its operation,such as how frequently it is controlled or triggered. In someembodiments, an accessory is represented as a favorite accessory basedon its operation with respect to a particular device or user profile,such as how frequently it is controlled by a particular device or userassociated with the particular device.

In the illustrated embodiment, representation 620-1 includes anaffordance. In FIG. 6B, device 600 detects a user input 630-1 (e.g., afinger tap) on representation 620-1 corresponding to a selection of thethermostat accessory. At the time of user input 630-1, the thermostataccessory is in a first state of heading to 72 degrees. In someembodiments, device 600 sends instructions to set the thermostataccessory to a second state in response to user input 630-1.

In some embodiments, device 600 determines a duration of the user input630-1 (e.g., the duration of a contact on touch-sensitive display 602).In some embodiments, device 600 determines a characteristic intensity ofthe user input (e.g., a characteristic intensity of a contact ontouch-sensitive display 602). Device 600 determines whether user input630-1 meets input criteria. In some embodiments, the input criteriainclude whether the duration of the user input exceeds a predeterminedamount of time and/or whether the characteristic intensity of the userinput exceeds a predetermined intensity.

In accordance with a determination that user input 630-1 does not meetthe input criteria, device 600 sends instructions to set the state ofthe thermostat accessory to a second state. In some embodiments, therepresentation of an accessory includes an indication of the state towhich the accessory will be set in response to user input that does notmeet the input criteria. For example, representation 620-2 in FIG. 6Aindicates that a fan accessory can be turned ON by selectingrepresentation 620-2.

FIG. 6C illustrates an exemplary response when user input 630-1 does notmeet the input criteria. As indicated in FIG. 6C, device 600 sendsinstructions to turn off the thermostat and updates representation 620-1to indicate an OFF status. In some embodiments, device 600 outputs ahaptic to indicate that the state has been changed.

Optionally, in accordance with a determination that user input 630-1does not meet the input criteria, device 600 sends instructions toswitch the state of an accessory from a first of two predeterminedstates to the second of the two predetermined states (e.g., ON/OFF,open/closed, locked/unlocked). In one example of this technique, inresponse to another user input identical to user input 630-1, device 600sends instructions to set the thermostat back on to its previous state(e.g., Heating, Set to 72 degrees). In this way, a user can providerelatively short or light taps to quickly toggle the state of anaccessory between two predetermined states. In some embodiments, inaccordance with a determination that user input does not meet the inputcriteria, device 600 sends instructions to switch the state to anadjacent state in a predetermined sequence of states (e.g.,OFF→LOW→HIGH→OFF). In this way, a user can easily cycle though asequence of states.

Alternatively, in accordance with a determination that user input 630-1meets the input criteria, device 600 displays a control affordanceindicating the available states of the accessory. FIG. 6D illustrates anexemplary user interface screen 610-2 with control affordance 608indicating the current state of the accessory (set to 72 degrees) andavailable states of the accessory. User interface screen 610-2 alsoincludes affordance 622 that, when selected, displays information aboutthe corresponding accessory and, optionally, displays affordances forchanging the information associated with the accessory. Informationassociated with the accessory optionally includes name, room, type,manufacturer, and a representative icon. Optionally, user interfacescreen 610-2 is partially translucent such that Home screen 610-1 ispartly visible.

With respect to control affordance 698, the horizontal bars on controlaffordance 608 represent temperatures to which the thermostat can beset, e.g., in one degree increments. In response to detecting user inputon control affordance 608 corresponding to selection of a particularstate, device 600 sends instructions to set the state of the accessoryto the selected state. In this way, a user can interact with controlaffordance 608 to select a state that is not available in response to auser input that does not meet the input criteria or to go directly to astate that is not adjacent to the current state. In FIG. 6D, device 600detects user input 630-2 (e.g., a tap), and in response, sets thethermostat to 75 degrees. Optionally, the thermostat can be changed to75 degrees by a vertical swipe from bar 614 to bar 616 on controlaffordance 608.

Optionally, in response to detecting user input 631-2 on controlaffordance 608 corresponding to selection of a particular state, thedevice updates the visual appearance of control affordance 608 toindicate that the accessory is set to the selected state, as shown forexample in FIG. 6E. Optionally, device 600 ceases to display controlaffordance 608 in response to selection of a state (e.g., device 600returns to user interface screen 610-1). In some embodiments, device 600ceases to display control affordance 608 in response to detecting userinput corresponding to selection of a location on the touch-sensitivedisplay 602 that does not correspond to the control affordance 608(e.g., area 612).

In some embodiments, device 600 updates the appearance of controlaffordance 608 to indicate the selected state but does not sendinstructions to set the state of the accessory until further user inputis detected. In some embodiments, device 600 sends instructions to setthe accessory to the selected state in response to selection of anotheraffordance (e.g., a “Set” or “Done” affordance) or an input on thetouch-sensitive display 602 located outside of the responsive area ofthe control affordance 608 (e.g., area 612).

Control affordance 608 is an example of a control affordance for anaccessory that operates in discrete states (e.g., a digital thermostatthat is adjustable in increments of one degree). FIGS. 6F-6I illustratesother examples of control affordances for accessories that operate indiscrete states. More specifically, FIG. 6F illustrates an exemplarycontrol affordance for a fan accessory that can operate in four discretestates; FIG. 6G illustrates an exemplary control affordance for a homesecurity system that can operate in various discrete alarm modes; andFIGS. 6H-6I illustrate an exemplary control affordance for a door orother lockable accessory with two discrete states. In some embodiments,device 600 displays the time at which the current state of an accessorywas set, as shown in the example of FIG. 6H (e.g., Unlocked at 5:00 PM).

In some embodiments, the available states of an accessory arecontinuous. FIG. 6J illustrates an exemplary control affordance 621 fora dimmable light switch that can operate over a continuous range ofbrightness. In some embodiments, a particular state is selected by,e.g., either tapping a corresponding position on affordance 621 (asindicated by element 630-3) or vertically dragging the top of bar 623until the desired state is obtained (as indicated by arrow 630-4).

In some embodiments, the accessory is a color-adjustable light bulb thatis configured to operate in different color states. The left side ofFIG. 6K illustrates a control affordance 625 for a color-adjustablelight bulb similar to control affordance 621 in FIG. 6J. But anaffordance 627 is also displayed. Selection of affordance 627 providesthe control affordance 629 for selecting the operating color of thecolor-adjustable light bulb, as illustrated on the right side of FIG.6K. Marker 618 indicates the current color of the bulb. In someembodiments, touching a position on control affordance 629 selects adifferent color.

In some embodiments, a relationship exists between two or moreaccessories such that when one accessory is controlled or operated,another related accessory is controlled. In some embodiments, inresponse to device 600 sending instructions to change the state of afirst accessory, a related second accessory is automatically controlledin addition to the first accessory. For example, an instruction tooperate a camera on the front porch of a location automatically turns ona microphone and/or speaker on the porch. In some embodiments, inresponse to a first accessory being activated, a related secondaccessory is automatically controlled. For example, when a visitor ringsa doorbell at the front door, a front door lock, porch camera, andintercom system associated with the front porch are automaticallyoperated (e.g., set to designated state).

In some embodiments, when an accessory is activated, device 600 displaysa notification indicating that the accessory has been activated. In someembodiments, the notification includes information regarding accessoriesrelated to the activated accessory and/or affordances for controllingaccessories related to the activated accessory. In one example, when thedoorbell is rung, device 600 displays a notification including an imagecaptured by a camera on the front porch and affordances to activate anintercom system at the front door, unlock the front door, and turn on aporch light. Exemplary embodiments of notifications corresponding tothis example are described in greater detail below with reference toFIGS. 34I-34K.

In some embodiments, the relationship between accessories is designatedby the user (e.g., by user input) to explicitly tie the functions of twoor more devices together—either triggering each to function togetherwhen one is triggered manually or through an automatic trigger (e.g.,time of day, sensor, etc.). In some embodiments, the relationship isestablished automatically based on the positions of the accessorieswithin a location (e.g., a relationship is automatically establishedbetween certain accessories because they are associated with a commonarea or room of a location). Similarly, the devices that are associatedbased on location function together when one is triggered manually orthrough an automatic trigger (e.g., time of day, sensor, etc.).

In some embodiments, a relationship between accessories is establishesbased on the type of accessory. In one example, a television, sound bar,AppleTV, and a home theater amplifier are automatically related becausethey are associated with an entertainment system. In this example,selection of an affordance to turn off the television optionally causesthe other accessories (sound bar, AppleTV, and home theater amplifier)to be powered off. In some embodiments, device 600 displays anaffordance, that may be automatically generated by the system,representing a group of accessories having the same type. Referringagain to the entertainment system example, device 600 optionallydisplays an affordance labeled “Entertainment System” that controls allof the accessories of the system listed above.

Referring back to FIG. 6A, Home screen 610-1 also includes arepresentation 624 of a person (e.g., a personal contact, friend, familymember) associated with the location. In some embodiments, device 600receives data indicating that a remote electronic device (e.g., asmartphone, tablet) associated with the person is present at thelocation. In some embodiments, representation 624 is displayed on Homescreen 610-1 in response to receiving the data indicating that theremote electronic device is present at the location. In the illustratedembodiment, representation 624 includes an image of the person. In someembodiments, the representation includes the person's name, initials, adesignated image associated with the person, or other text, image,video, or the like that is associated with the person. In someembodiments, a person associated with a device that is not present atthe location or that has an unknown position (e.g., away or inactive)are not represented on Home screen 610-1. In some embodiments, Homescreen 610-1 includes one or more representations of persons who aredetermined not to be present at the location.

In some embodiments, Home screen 610-1 includes an indication of thestatus or position of the person. In some embodiments, the status orposition is relative to the location. For example, in some embodiments,Home screen 610-1 includes an indication of whether the person isdetermined to be present at the location or away from the location orwhether the status of the person is determined to be unknown orinactive. In some embodiments, a status of away or unknown is indicatedby shading or greying out the representation of the person. In someembodiments, the status of the person is determined based on thelocation of a device associated with the person. The status optionallyalso indicates whether the person has been invited to be associated withthe location for the purposes of accessing information about location.

Home screen 610-1 also includes an indication 626 of a status of thelocation. In some embodiments, the status includes an overall assessmentof the location (e.g., “All safe and sound”). In the embodiment shown inFIG. 6A, indication 626 includes the status of accessories associatedwith the location (e.g., Garage door open; thermostat at 70 degrees nowheating to 72 degrees; kitchen fan ON). In some embodiments, the statusof an accessory is categorized (e.g., as Urgent, Attention, orNon-critical) and displayed based on criticality with more criticalstatus displayed above less critical status. Examples of an Urgentstatus can include activation of a security alarm, smoke detector, COdetector, or an indication that a hub that provides the status of anaccessory is not working. Examples of an Attention status can include anopen or unlocked door or window or that an accessory is disconnected orunavailable. Examples of non-critical status can include that a light ison and the state of a thermostat.

Home screen 610-1 also includes a representation 640-1 of a sceneprofile. A scene profile includes data regarding designated states ofone or more accessories at the location. A scene profile may also bereferred to simply as a “scene.” In FIG. 6A, the scene profilecorresponding to representation 640-1 is designated as a “FavoriteScene”. In some embodiments, a scene profile is designated based on oneor more of the following: (1) a user input explicitly designating thescene profile, (2) the frequency with which the scene profile isapplied, (3) how recently the scene profile was applied, and (4) howfrequently or recently accessories associated with the scene profilehave been selected, controlled, or triggered. Similar to designatedaccessories described above, in some embodiments, a scene profile isdesignated based on actions taken by a particular device or userassociated with a device (e.g., which scene profiles are frequentlyselected by a user).

In the embodiment of FIG. 6A, representation 640-1 includes anaffordance. In some embodiments, in response to detecting user inputcorresponding to selection of representation 640-1, device 600 sendsinstructions to apply the designated states of the accessories includedin the corresponding scene profile. In some embodiments, in response todetecting user input corresponding to selection representation 640-1 ofa scene profile, device 600 displays user interface screen 610-3 shownin FIG. 6L, which includes representations of the accessories associatedwith the scene profile. The representations of the plurality ofaccessories in user interface screen 610-3 include indications of thedesignated states of the respective accessories (e.g., the state towhich the accessory is set when the scene profile is implemented). Insome embodiments, the indication of the designated state includes theaction to be taken when the scene profile is implements (e.g., turnoff).

Home screen 610-1 also includes affordance 628. In response to detectinguser input on affordance 628, device 600 displays an option 630 tocreate an additional scene profile, as shown for example in FIG. 6M. Insome embodiments, device 600 displays the option to create an additionalscene profile in response to other types of user input.

In some embodiments, Home screen 610-1 includes an image captured by acontrollable external camera the location. In the embodiment shown inFIG. 6N, Home screen 610-1 is scrollable and is scrolled by knowntechniques to reveal area 632 for display of an image included in a livevideo feed from a controllable external camera in the kitchen of thelocation. In some embodiments, the image is from a single point in time.In some embodiments, the image is a live video feed. In someembodiments, the image is a single image that can be selected (e.g.,tapped) to display a live video feed from the controllable externalcamera. In some embodiments, one or more affordances or camera imagesmay be off screen and may be displayed with an input, such as a buttonor scrolling gesture such as a swipe. In some embodiments, displayingthe one or more affordances or camera images that were initially offscreen displaces one or more affordances such that they are no longerdisplayed.

Device 600 also displays a Rooms tab 604-2. In response to detectinguser input (e.g., a tap) on Rooms tab 604-2, device 600 displays arepresentation of at least one room associated with the location, asshown, for example, on user interface screen 610-4 in FIG. 6O. In theembodiment of FIG. 6O, the representations of the rooms include imagesassociated with the corresponding rooms. In response to detecting userinput selecting a room on user interface 610-4, device 600 displaysrepresentations of accessories associated with the selected room and,optionally, representations of scene profiles including accessoriesassociated with the selected room. FIG. 6P illustrates an exemplary userinterface screen 610-5 corresponding to the Bedroom with representationsof accessories and scene profiles associated with the Bedroom. In someembodiments, the representations of the accessories indicate the currentstates of the accessories. In some embodiments, an image associated withthe room is displayed as wallpaper on user interface screen 610-5 behindthe representations of the accessories. In some embodiments, therepresentations of the accessories associated with the selected room areaffordances for controlling the accessories or accessing user interfacescreens for controlling the corresponding accessories as described abovewith reference to representation 620-1 and FIGS. 6B-6K.

User interface screen 610-1 also includes an Automation tab 604-3. Inresponse to detecting user input on Automation tab 604-3, device 600displays a representation of an automation profile and/or an option tocreate an automation profile. An automation profile includes datarepresenting criteria for controlling a state of at least one accessoryat a location and designated states for the at least one accessory. Inother words, an automation profile includes the states to which selectedaccessories are set when the criteria of the automation profile is met.An “automation profile” may be referred to simply as an “automation”.Setting the accessories associated with the automation profile to thedesignated states in response to determining that the criteria of theautomation profile is met is sometimes referred to as implementing theautomation profile. An automation profile can include data for a sceneprofile, which also includes data regarding a state of an accessory.

FIG. 6Q illustrates an exemplary user interface screen 610-6 displayedin response to detecting user input on Automation tab 604-3. Userinterface screen 610-6 includes representation 634 of an automationprofile named “When I leave home” associated with the location.Representation 634 also includes an indication of the scenes and/oraccessories associated with the automation profile, “Run Scene 1” and“Control 3 accessories”. In some embodiments, the name of the automationprofile is indicative of the criteria of the automation profile. In someembodiments, the representation of the automation profile includes anaffordance that when selected displays representations of theaccessories and/or scene profiles associated with the automationprofile. In some embodiments, the representation of an accessoryassociated with an automation profile includes an affordance that whenselected changes the designated state of the accessory for theautomation profile or provides a user interface for selecting adesignated state of the accessory for the automation profile. In someembodiments, the designated state of an accessory associated with anautomation profile is selected by applying the techniques described withrespect to FIGS. 6B-6K.

Device 600 also provides the ability to view information associated witha different location and/or create a location profile for an additionallocation. In the embodiment shown in FIG. 6R, device 600 detects userinput on affordance 636 included on Home screen 610-1. In response todetecting selection of affordance 636, device 600 displays menu 638,which provides options to view information associated with a differentlocation (e.g., Lisa's Home or Beach House) and add an additionallocation (e.g., Add Location). Device 600 also provides an option (e.g.,Location Details . . . ) to view and or change profile informationassociated with the location, such as the location profile informationdescribed above.

Device 600 also provides the ability to add an accessory associated withthe location. Referring again to FIG. 6M, device 600 detects user inputon affordance 628 included on Home screen 610-1, and in response,displays option 642 to associate an accessory with the location.

In some embodiments, device 600 displays a People tab 604-4, as shown inFIG. 6S. In response to detecting a user input on People tab 604-4,device 600 displays representations of people associated with thelocation. In the illustrated embodiments, the representations 644-1,644-2, and 644-3 of the people are displayed on user interface screen610-7. In the illustrated embodiment, the representations of the peopleinclude images of the people. In some embodiments, the representationsinclude the person's name, initials, a designated image associated withthe person, or other text, image, video, or the like that is associatedwith the person. In some embodiments, user interface screen 610-7includes representation of all the people associated with the location.In some embodiments, people with a status indicating that they are notpresent at the location or that their position is unknown (e.g., away,inactive, etc.) are not displayed (e.g., only representations of peopledetermined to be present at the location are displayed). In someembodiments, one or more representations of persons who are determinednot to be present at the location are displayed.

In FIG. 6S, user interface screen 610-7 includes indications of thestatus or position of the people. In some embodiments, the status orposition is relative to the location. For example, in some embodiments,an indication of whether the person is determined to be present at thelocation or away from the location or whether the status of the personis determined to be unknown or inactive is displayed. In someembodiments, the status of the person is determined based on thelocation of a device associated with the person. In some embodiments,speaker identification using one or more microphones may be used todetermine the presence and identity of an individual. The status mayalso indicate whether the person has been invited to be associated withthe location for the purposes of accessing information about location(e.g., Invited).

FIGS. 7A-7C depict a flow diagram illustrating a method for managingcontrollable external devices using an electronic device in accordancewith some embodiments. Method 700 is performed at a device (e.g., 100,300, 500, or 600) with a display and, optionally, a touch-sensitivesurface. Some operations in method 700 are, optionally, combined, theorder of some operations is, optionally, changed, and some operationsare, optionally, omitted.

As described below, method 700 provides an intuitive way for managingcontrollable external devices. The method reduces the cognitive burdenon a user for managing controllable external devices, thereby creating amore efficient human-machine interface. For battery-operated computingdevices, enabling a user to manage controllable external devices fasterand more efficiently conserves power and increases the time betweenbattery charges.

At block 702, the device displays a user interface screen (e.g., 610-1)including an indication of a location and a representation of acontrollable external (e.g., 620-1) device at the location.

Optionally, at block 704, the device receives data indicating that aremote electronic device associated with a person is present at thelocation, and in response to receiving the data indicating that theremote electronic device is present at the location, displays, on theuser interface screen, a representation of the person associated withthe remote electronic device (e.g., 624).

Optionally, at block 706, the device displays, on the user interfacescreen, an indication of a status of the location (e.g., 626).

Optionally, at block 708, the device displays, on the user interfacescreen, a representation of a scene profile (e.g., 640-1), where thescene profile includes data regarding designated states of a pluralityof controllable external devices at the location; detects a first userinput corresponding to selection of the scene profile; and in responseto detecting the first user input, displays a representation of theplurality of controllable external devices associated with the sceneprofile (e.g., 910-2).

Optionally, at block 710, the device displays, on the user interfacescreen, an image captured by a controllable external camera at thelocation (e.g., 632).

At block 712, the device detects a second user input (e.g., 630-1)corresponding to selection of the controllable external device.

At block 714, in response to detecting the second user input, the devicedisplays a user interface object (e.g., 608) indicating a plurality ofpossible states of the controllable external device.

At block 716, while the user interface object indicating the pluralityof possible states of the controllable external device is displayed, thedevice detects a third user input (e.g., 630-2) corresponding toselection of a designated state of the controllable external device.

At block 718, in response to detecting the third user input, the devicesends instructions to set the controllable external device to thedesignated state.

Optionally, at block 720, the device detects a fourth user input whiledisplaying the user interface screen; in response to detecting thefourth user input, displays representations of at least one designatedroom associated with the location (e.g., FIG. 6O); detects a fifth userinput corresponding to selection of a first designated room of the atleast one designated room; and in response to detecting the fifth userinput, displays a representation of at least one controllable externaldevice associated with the first designated room (e.g., FIG. 6P).

Optionally, at block 722, the device detects a sixth user input whiledisplaying the user interface screen; and in response to detecting thesixth user input, displays a representation of an automation profileand/or an option to create the automation profile (e.g., FIG. 6Q), wherethe automation profile includes data representing criteria forcontrolling a state of at least one controllable external device at thelocation and designated states for the at least one controllableexternal device.

Optionally, at block 724, the device detects a seventh user input whiledisplaying the user interface screen; and in response to detecting theseventh user input, displays representations of people associated withthe location (e.g., FIG. 6S).

Optionally, at block 726, the device detects an eighth user input whiledisplaying the user interface screen; and in response to detecting theeighth user input, displays an option to perform one or more of:creating an additional scene profile, viewing information associatedwith a different location and/or add an additional location, changingprofile information of the location, and associating a secondcontrollable external device with the location (e.g., FIGS. 6M (630) and6R (638)).

Note that details of the processes described above with respect tomethod 700 (e.g., FIGS. 7A-7C) are also applicable in an analogousmanner to the methods described below. For example, methods 1000, 1300,1600, 1900, 2100, 2300, 2600, 2900, and 3200 optionally include one ormore of the characteristics of the various methods described above withreference to method 700.

In accordance with some embodiments, FIG. 8 shows an exemplaryfunctional block diagram of an electronic device 800 configured inaccordance with the principles of the various described embodiments. Inaccordance with some embodiments, the functional blocks of electronicdevice 800 are configured to perform the techniques described above. Thefunctional blocks of the device 800 are, optionally, implemented byhardware, software, or a combination of hardware and software to carryout the principles of the various described examples. It is understoodby persons of skill in the art that the functional blocks described inFIG. 8 are, optionally, combined or separated into sub-blocks toimplement the principles of the various described examples. Therefore,the description herein optionally supports any possible combination orseparation or further definition of the functional blocks describedherein.

As shown in FIG. 8, an electronic device 800 includes a display unit 802configured to display a graphic user interface, optionally, atouch-sensitive surface unit 804 configured to receive contacts, and aprocessing unit 806 coupled to the display unit 802 and, optionally, thetouch-sensitive surface unit 804. In some embodiments, the processingunit 806 includes a display enabling unit 808, a detecting unit 810, acausing unit 812, and receiving unit 814.

The processing unit 806 is configured to: enable display (e.g., withdisplay enabling unit 808) of, on a first user interface screen, anindication of a location; enable display (e.g., with display enablingunit 808) of, on the first user interface screen, a representation of acontrollable external device at the location; detect (e.g., withdetecting unit 810) a first user input corresponding to selection of thecontrollable external device; in response to detecting the first userinput, enable display (e.g., with display enabling unit 808) of a userinterface object indicating a plurality of possible states of thecontrollable external device; while the user interface object indicatingthe plurality of possible states of the controllable external device isdisplayed, detect (e.g., with detecting unit 810) a second user inputcorresponding to selection of a designated state of the controllableexternal device; and in response to detecting the second user input,cause (e.g., with causing unit 812) sending of instructions to set thecontrollable external device to the designated state.

In some embodiments, the processing unit 806 is further configured to:receive (e.g., with receiving unit 814) data indicating that a remoteelectronic device associated with a person is present at the location;and in response to receiving the data indicating that the remoteelectronic device is present at the location, enable display (e.g., withdisplay enabling unit 808) of, on the first user interface screen, arepresentation of the person associated with the remote electronicdevice.

In some embodiments, the processing unit 806 is further configured to:enable display (e.g., with display enabling unit 810) of, on the firstuser interface screen, an indication of a status of the location.

In some embodiments, the processing unit 806 is further configured to:enable display (e.g., with display enabling unit 808) of, on the firstuser interface screen, a representation of a scene profile, where thescene profile includes data regarding designated states of a pluralityof controllable external devices at the location; detect (e.g., withdetecting unit 810) a third user input corresponding to selection of thescene profile; and in response to detecting the third user input, enabledisplay (e.g., with display enabling unit 808) of a representation ofthe plurality of controllable external devices associated with the sceneprofile. In some embodiments, the processing unit 806 is furtherconfigured to: while the first user interface screen is displayed,detect (e.g., with detecting unit 810) a fourth user input; and inresponse to detecting the fourth user input, enable display (e.g., withdisplay enabling unit 808) of an option to create an additional sceneprofile.

In some embodiments, the processing unit 806 is further configured to:enable display (e.g., with display enabling unit 808) of, on the firstuser interface screen, an image captured by a controllable externalcamera at the location, where the image captured by the controllableexternal camera is included in a live video feed from the controllableexternal camera.

In some embodiments, the processing unit 806 is further configured to:while the first user interface screen is displayed, detect (e.g., withdetecting unit 810) a fifth user input; in response to detecting thefifth user input, enable display (e.g., with display enabling unit 808)of representations of at least one designated room associated with thelocation; detect (e.g., with detecting unit 810) a sixth user inputcorresponding to selection of a first designated room of the at leastone designated room; and in response to detecting the sixth user input,enable display (e.g., with display enabling unit 808) of arepresentation of at least one controllable external device associatedwith the first designated room.

In some embodiments, the processing unit 806 is further configured to:while the first user interface screen is displayed, detect (e.g., withdetecting unit 810) a seventh user input; and in response to detectingthe seventh user input, enable display (e.g., with display enabling unit808) of a representation of an automation profile and/or an option tocreate the automation profile, where the automation profile includesdata representing criteria for controlling a state of at least onecontrollable external device at the location and designated states forthe at least one controllable external device.

In some embodiments, the processing unit 806 is further configured to:while the first user interface screen is displayed, detect (e.g., withdetecting unit 810) an eighth user input; and in response to detectingthe eighth user input, enable display (e.g., with display enabling unit808) of an option to view information associated with a differentlocation and/or add an additional location.

In some embodiments, the processing unit 806 is further configured to:while the first user interface screen is displayed, detect (e.g., withdetecting unit 810) a ninth user input; and in response to detecting theninth user input, enable display (e.g., with display enabling unit 808)of an option to change profile information of the location.

In some embodiments, the processing unit 806 is further configured to:while the first user interface screen is displayed, detect (e.g., withdetecting unit 810) a tenth user input; and in response to detecting thetenth user input, enable display (e.g., with display enabling unit 808)of an option to associate a second controllable external device with thelocation.

In some embodiments, the processing unit 806 is further configured to:while the first user interface screen is displayed, detect (e.g., withdetecting unit 810) a eleventh user input; and in response to detectingthe eleventh user input, enable display (e.g., with display enablingunit 808) of representations of people associated with the location.

The operations described above with reference to FIGS. 7A-7C are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG. 8.For example, displaying operations 702, 704, 706, 708, 710, 714, 720,722, 724, and 726, detecting operations 708, 712, 716, 720, 722, 724,and 726, sending operation 718, and receiving operation 704 are,optionally, implemented by event sorter 170, event recognizer 180, andevent handler 190. Event monitor 171 in event sorter 170 detects acontact on touch-sensitive display 112, and event dispatcher module 174delivers the event information to application 136-1. A respective eventrecognizer 180 of application 136-1 compares the event information torespective event definitions 186, and determines whether a first contactat a first location on the touch-sensitive surface corresponds to apredefined event or sub event, such as activation of an affordance on auser interface. When a respective predefined event or sub-event isdetected, event recognizer 180 activates an event handler 190 associatedwith the detection of the event or sub-event. Event handler 190optionally utilizes or calls data updater 176 or object updater 177 toupdate the application internal state 192. In some embodiments, eventhandler 190 accesses a respective GUI updater 178 to update what isdisplayed by the application. Similarly, it would be clear to a personhaving ordinary skill in the art how other processes can be implementedbased on the components depicted in FIGS. 1A-1B.

FIGS. 9A-9F illustrate exemplary user interfaces for creating and/orconfiguring scene profiles for controllable external devices, inaccordance with some embodiments. The user interfaces in these figuresare used to illustrate the processes described below, including theprocesses in FIG. 10.

FIG. 9A illustrates user interface screen 910-1 for creating a new sceneprofile displayed on device 600. In some embodiments, user interfacescreen 910-1 is displayed in response to selection of the “New Scene”option shown in FIG. 6M. User interface screen 910-1 prompts the user tocreate a new scene profile and/or provides one or more predefined (e.g.,suggested) scene profiles. A predefined scene profile includes dataregarding predetermined states of predetermined controllable electronicdevices at a location. User interface screen 910-1 includes anaffordance 904 for creating a new custom scene profile andrepresentations 902-1, 902-2, 902-3, and 902-4 of exemplary predefinedscene profiles named “Good Morning”, “Leave Home”, “Arrive Home”, and“Good Night”.

FIG. 9B illustrates device 600 displaying user interface screen 910-2corresponding to a new custom scene profile named Movie Night. Inaddition to the name of the scene profile, user interface screen 910-2also includes an image 905 (e.g., a glyph, symbol, picture) associatedwith the scene profile. In some embodiments, a user can manually selectthe image associated with the scene profile. User interface screen 910-2also includes affordance 906 labeled “Add Accessories” for addingaccessories to the scene profile. In response to user input selectingaffordance 906, device 600 displays user interface screen 910-3 (FIG.9C), which includes representations 908-1 to 908-5 of accessories thatare situated at the location. Representations 908-1 to 908-5 arearranged based on the corresponding rooms to which they are associated.The representations include indications of whether the correspondingaccessory is associated with the scene profile. In the illustratedexample, the circle in the upper right corner of each representation908-1 to 908-5 is checked or filled-in if the corresponding accessory isassociated with the scene profile. Otherwise, the circle is blank. Otherexamples of indications that an accessory is included in the sceneprofile include highlighting, bolding, or the like.

In the embodiment of FIG. 9C, representations 908-1 to 908-5 areaffordances. Device 600 detects user input 920-1 corresponding to aselection of a first accessory (e.g., a tap on representation 908-1).After detecting user input 920-1, device 600 associates the firstaccessory with the scene profile. FIG. 9D shows an updated version ofuser interface screen 910-3 indicating that the accessory correspondingto representation 908-1 is associated with the scene profile. Device 600associates the first accessory with the scene profile by including dataidentifying the first accessory and a first state of the first accessoryin the scene profile. If the scene profile already exists, thenincluding the data in the scene profile includes updating the existingprofile (e.g., adding the data to the existing scene profile). In someembodiments, if the scene profile does not yet exist when the accessoryis selected, then including the data in the scene profile includescreating a new scene profile with the data.

In response to detecting selection of affordance 914, device 600 causesall of the accessories associated with the Bedroom to be included in thescene profile. Affordance 916 provides analogous functionality withrespect to the Bathroom. In some embodiments, user interface screen910-3 includes an affordance that when selected causes all of theaccessories on user interface screen to be included in the sceneprofile.

In some embodiments, the representations 908-1 to 908-5 on userinterface screen 910-3 include indications of the current states of thecorresponding accessories. In some embodiments, device 600 includes inthe scene profile the current state of the accessory at the time theaccessory is selected. In some embodiments, if an accessory is alreadyassociated with the scene profile, the representation on user interfacescreen 910-3 corresponding to the accessory indicates the state of theaccessory included in the scene profile, which is not necessarily thesame as the current state of the accessory. In some embodiments, therepresentations 908-1 to 908-5 include indications of the states thatwill be included in the scene profile if the accessory is selected,which may be, for example, the current state of the accessory or a statedesignated by a user.

In some embodiments, device 600 detects second user input 920-2corresponding to selection of a second accessory (e.g., a contact at aposition on the touch-sensitive display 602 corresponding to therepresentation 908-5). Device 600 determines whether second user input920-2 meets threshold criteria. In some embodiment, threshold criteriainclude, e.g., a threshold duration of time of a contact or a thresholdcharacteristic intensity of the contact.

In response to detecting the second user input 920-2 and in accordancewith a determination that the threshold criteria are met, device 600displays an indication of a first state of the second accessory. In someembodiments, the indication of the first state is displayed as part of acontrol affordance such as those illustrated in FIGS. 6C-6K. In theembodiment illustrated in FIG. 9C, the accessory corresponding toselected representation 908-5 is a light. In some embodiments, thecontrol affordance displayed in FIG. 6J is displayed so that the usercan select state of the light.

Device 600 detects an input corresponding to a selection of the firststate (e.g., a tap on the control affordance in FIG. 6J). Afterdetecting selection of the first state, device 600 includes dataidentifying the second accessory and the first state of the secondaccessory in the scene profile. In some embodiments, after detecting theselection of the first state, device 600 displays representation 908-5of the second accessory with an indication of the first state (e.g.,OFF) and that the accessory has been added to the scene profile, asshown in FIG. 9D. In some embodiments, device 600 includes the dataidentifying the first state in the scene profile directly in response touser input 920-2 upon determining that the threshold criteria are met.In some embodiments, device 600 indicates that the designated state ofthe accessory for the scene has been selected, e.g., by outputting ahaptic.

FIG. 9E illustrates an updated version of user interface screen 910-2 ondevice 600 with a summary of the scene profile. Device 600 displays userinterface screen 910-2 in response to detecting user input, e.g.,selection of the affordance labeled “Done” on user interface screen910-3. User interface screen 910-2 includes representations 908-6 and908-7 of the accessories associated with the scene profile.

Updated user interface screen 910-2 also includes options (e.g.,affordances) to add or remove the data associated with the accessoryfrom the scene profile (e.g., 912-1), test the scene profile (e.g.,912-2), designate the scene profile (e.g., 912-3), and delete the sceneprofile (e.g., 912-4). In some embodiments, testing the scene profileincludes sending instructions to implement (at least temporarily) thedesignated states of the accessories included in the scene profile. Insome embodiments, a designated scene is displayed as a favorite scene onthe Home screen 610-1.

Returning now to FIG. 9A, in response to detecting user inputcorresponding to selection of one of the predefined scene profiles 902-1to 902-4, device 600 displays a user interface screen includingrepresentations of predetermined accessories at the location. FIG. 9Fillustrates an exemplary user interface screen corresponding to thepredefined scene profile Good Night 902-4. As shown in FIG. 9F, the GoodNight scene profile is pre-populated with two predetermined accessorieswith predetermined states. More specifically, the front door is set tobe locked and the thermostat in the Bedroom is set to be turned down to68 degrees. Additional exemplary embodiments related to predefined sceneprofiles are described below.

In some embodiments, the predetermined accessories are defaultaccessories based on the selected predefined scene profile that areautomatically included in the scene profile without requiring userselection (e.g., the Garage door may be predetermined for the “ArriveHome” predefined scene). In some embodiment, the predeterminedaccessories are determined based on actions previously performed bydevice 600 (or a user profile associated with device 600) with respectto the predetermined accessories. In some embodiments, an accessory issuggested based on, e.g., frequency of use or correlation between anevent or time and use of an accessory (e.g., a bedroom light isdetermined for the “Good Morning” scene profile if device 600 typicallyoperates the light in the bedroom in the morning or the dishwasher maybe predetermined for the “Good Night” scene profile if it is commonlyoperated after 11 pm).

In some embodiments, the representations of the predeterminedaccessories include indications of predetermined states (e.g., suggestedstates) for the predetermined accessories, as shown in the embodiment ofFIG. 9F. In some embodiments, the predetermined states are defaultstates based on the selected predefined scene profile that areautomatically included in the scene profile without requiring userselection (e.g., the suggested status of the Garage Door for “ArriveHome” is “open”).

In some embodiments, the predetermined states of the predeterminedaccessories are determined based on actions associated with a user ofdevice 600 previously taken with respect to the predeterminedaccessories. In some embodiments, a state of an accessory is suggestedbased on, e.g., frequency of selection or correlation between an eventor time and state of an accessory (e.g., the bedroom light set to ON forthe “Good Morning” scene profile if device 600 typically turns on thelight in the bedroom in the morning or the state of the dishwasher maybe predetermined to “start” for the “Good Night” scene profile if it iscommonly started after 11 pm).

FIG. 10 is a flow diagram illustrating a method for managingcontrollable external devices using an electronic device in accordancewith some embodiments. Method 1000 is performed at a device (e.g., 100,300, 500, or 600) with a display and, optionally, a touch-sensitivesurface. Some operations in method 1000 are, optionally, combined, theorder of some operations is, optionally, changed, and some operationsare, optionally, omitted.

As described below, method 1000 provides an intuitive way for managingcontrollable external devices. The method reduces the cognitive burdenon a user for managing controllable external devices, thereby creating amore efficient human-machine interface. For battery-operated computingdevices, enabling a user to manage controllable external devices fasterand more efficiently conserves power and increases the time betweenbattery charges.

Optionally, at block 1002, the device displays a first user interfacescreen (e.g., 910-1) including a representation of a predefined sceneprofile (e.g., 902-1 to 902-4), where the predefined scene profileincludes data regarding predetermined states of predeterminedcontrollable electronic devices at a location.

Optionally, at block 1004, the device, while displaying the first userinterface screen, detects a first user input corresponding to selectionof the predefined scene profile; and in response to detecting the firstuser input, displays a second user interface screen (e.g., FIG. 9F)including a plurality of representations of the predeterminedcontrollable external devices at the location. Optionally, the pluralityof representations of the predetermined controllable external devices atthe location includes indications of the predetermined states for thepredetermined controllable external devices. Optionally, thepredetermined controllable external devices are determined based onactions previously performed by the device with respect to thepredetermined controllable external devices. Optionally, thepredetermined states of the predetermined controllable external devicesare determined based on actions associated with a user of the electronicdevice previously taken with respect to the predetermined controllableexternal devices.

At block 1006, the device displays a first representation of a firstcontrollable external device, where the first controllable externaldevice is situated at the location (e.g., 908-1 to 908-5 in FIG. 9C).Optionally, the first representation of the controllable external deviceincludes an indication of the current state of the controllable externaldevice.

At block 1008, the device detects a second user input (e.g., 920-1)corresponding to a selection of the first controllable external device.

At block 1010, after detecting the second user input, the deviceincludes data identifying the first controllable external device and afirst state of the first controllable external device in a scene profile(e.g., FIG. 9D).

Optionally, at block 1012, the device detects a third user input, and inresponse to detecting the third user input, displays a third userinterface screen that includes a second representation of thecontrollable external device (e.g., 908-6). Optionally, the third userinterface screen includes an option to perform at least one of removingthe data associated with the controllable external device from the sceneprofile (e.g., 912-1), testing the scene profile (e.g., 912-2),designating the scene profile (e.g., 912-3), and deleting the sceneprofile (e.g., 912-4).

Optionally, at block 1014, the device: displays a representation of asecond controllable external device situated at the location (e.g.,908-5); detects a fourth user input (e.g., 920-2) corresponding toselection of the second controllable external device; determines whetherthe fourth user input meets threshold criteria; in response to detectingthe fourth user input and in accordance with a determination that thethreshold criteria are met, displays an indication of a first state ofthe second controllable external device (e.g., 608); detects a fifthuser input corresponding to a selection of the first state of the secondcontrollable external device (e.g., 630-2); and after detecting thefifth user input, includes data identifying the second controllableexternal device and the first state of the second controllable externaldevice in the scene profile.

Note that details of the processes described above with respect tomethod 1000 (e.g., FIG. 10) are also applicable in an analogous mannerto the methods described above and below. For example, methods 700,1300, 1600, 1900, 2100, 2300, 2600, 2900, and 3200 optionally includeone or more of the characteristics of the various methods describedabove with reference to method 1000.

In accordance with some embodiments, FIG. 11 shows an exemplaryfunctional block diagram of an electronic device 1100 configured inaccordance with the principles of the various described embodiments. Inaccordance with some embodiments, the functional blocks of electronicdevice 1100 are configured to perform the techniques described above.The functional blocks of the device 1100 are, optionally, implemented byhardware, software, or a combination of hardware and software to carryout the principles of the various described examples. It is understoodby persons of skill in the art that the functional blocks described inFIG. 11 are, optionally, combined or separated into sub-blocks toimplement the principles of the various described examples. Therefore,the description herein optionally supports any possible combination orseparation or further definition of the functional blocks describedherein.

As shown in FIG. 11, an electronic device 1100 includes a display unit1102 configured to display a graphic user interface, optionally, atouch-sensitive surface unit 1104 configured to receive contacts, and aprocessing unit 1106 coupled to the display unit 1102 and, optionally,the touch-sensitive surface unit 1104. In some embodiments, theprocessing unit 1106 includes a display enabling unit 1108, a detectingunit 1110, an including unit 1112, and a determining unit 1114.

The processing unit 1106 is configured to: enable display (e.g., withdisplay enabling unit 1108) of a first representation of a firstcontrollable external device, where the first controllable externaldevice is situated at a location; detect (e.g., with detecting unit1110) a first user input corresponding to a selection of the firstcontrollable external device; and include (e.g., with including unit1112) data identifying the first controllable external device and afirst state of the first controllable external device in a scene profileafter detecting the first user input. In some embodiments, the firstrepresentation of the controllable external device includes anindication of the current state of the controllable external device.

In some embodiments, the processing unit 1106 is further configured to:enable display (e.g., with display enabling unit 1108) of arepresentation of a second controllable external device situated at thelocation; detect (e.g., with detecting unit 1110) a second user inputcorresponding to selection of the second controllable external device;determine (e.g., with determining unit 1114) whether the second userinput meets threshold criteria; in response to detecting the second userinput and in accordance with a determination that the threshold criteriaare met, enable display (e.g., with display enabling unit 1108) of anindication of a first state of the second controllable external device;detect (e.g., with detecting unit 1110) a third user input correspondingto a selection of the first state of the second controllable externaldevice; and after detecting the third user input, include (e.g., withincluding unit 1112) data identifying the second controllable externaldevice and the first state of the second controllable external device inthe scene profile.

In some embodiments, the processing unit 1106 is further configured to:detect (e.g., with detecting unit 1110) a fourth user input; and inresponse to detecting the fourth user input, enable display (e.g., withdisplay enabling unit 1108) of a first user interface screen thatincludes a second representation of the controllable external device. Insome embodiments, the first user interface screen includes an option toperform at least one of removing the data associated with thecontrollable external device from the scene profile, testing the sceneprofile, designating the scene profile, and deleting the scene profile.

In some embodiments, the processing unit 1106 is further configured toenable display (e.g., with display enabling unit 1108) of a second userinterface screen including a representation of a predefined sceneprofile prior to the first representation of the controllable externaldevice being displayed, where the scene profile includes data regardingpredetermined states of predetermined controllable electronic devices ata location.

In some embodiments, the processing unit 1106 is further configured todetect (e.g., with detecting unit 1110) a fifth user input correspondingto selection of the predefined scene profile while the second userinterface screen is displayed; and in response to detecting the fifthuser input, enable display (e.g., with display enabling unit 1108) of athird user interface screen including a plurality of representations ofthe predetermined controllable external devices at the location. In someembodiments, the plurality of representations of the predeterminedcontrollable external devices at the location includes indications ofthe predetermined states for the predetermined controllable externaldevices.

In some embodiments, the predetermined controllable external devices aredetermined based on actions previously performed by the electronicdevice with respect to the predetermined controllable external devices.In some embodiments, the predetermined states of the predeterminedcontrollable external devices are determined based on actions associatedwith a user of the electronic device previously taken with respect tothe predetermined controllable external devices.

The operations described above with reference to FIG. 10 are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.11. For example, displaying operations 1002, 1004, 1006, and 1014,detecting operations 1004, 1008, 1012, and 1014, including operations1010 and 1014, and determining operation 1014 are, optionally,implemented by event sorter 170, event recognizer 180, and event handler190. Event monitor 171 in event sorter 170 detects a contact ontouch-sensitive display 112, and event dispatcher module 174 deliversthe event information to application 136-1. A respective eventrecognizer 180 of application 136-1 compares the event information torespective event definitions 186, and determines whether a first contactat a first location on the touch-sensitive surface corresponds to apredefined event or sub event, such as activation of an affordance on auser interface. When a respective predefined event or sub-event isdetected, event recognizer 180 activates an event handler 190 associatedwith the detection of the event or sub-event. Event handler 190optionally utilizes or calls data updater 176 or object updater 177 toupdate the application internal state 192. In some embodiments, eventhandler 190 accesses a respective GUI updater 178 to update what isdisplayed by the application. Similarly, it would be clear to a personhaving ordinary skill in the art how other processes can be implementedbased on the components depicted in FIGS. 1A-1B.

FIGS. 12A-12C illustrate exemplary user interfaces for creating and/orconfiguring automation profiles for controllable external devices, inaccordance with some embodiments. The user interfaces in these figuresare used to illustrate the processes described below, including theprocesses in FIGS. 13 and 21.

FIG. 12A illustrates device 600 displaying user interface screen 1210-1for defining an automation profile associated with a location. In someembodiments, user interface screen 1210-1 is displayed in response toselection of the affordance in FIG. 6Q labeled “New Automation.” Userinterface screen 1210-1 includes representations 1202-1 to 1202-4 ofcriteria that may be selected for an automation profile. An automationprofile includes data representing at least one accessory, a designatedstate for each represented accessory, and automation criteria forcontrolling the at least one accessory. In some embodiments, anautomation profile includes a scene profile (or a portion thereof) thatis executed when the automation criteria of the automation profile ismet. The automation criteria represented by items 1202-1 to 1202-4 mayalso be referred to as a “triggering event”, which is an event that inresponse to which an automation profile is to be implemented.

In some embodiments, the automation criteria are based at least in parton the position of device 600 relative to the location. Examples oflocation-based criteria include, but are not limited to: whether theelectronic device is less than a threshold distance from the firstlocation (e.g., “I am at home”); whether the electronic device isgreater than a threshold distance from the first location (e.g., “I amaway from home”); whether the electronic device has moved from greaterthan a threshold distance from the first location to less than thethreshold distance from the location (e.g., “I arrive home” 1202-2); andwhether the electronic device has moved from less than a thresholddistance from the first location to greater than the threshold distancefrom the location (e.g., “I leave home” 1202-1). In some embodiments,the threshold distance is the same in all directions from the location(e.g., a circular boundary). In some embodiments, the threshold distancevaries with direction from the location.

In some embodiments, the automation criteria are based at least in parton a time (e.g., “A time of day occurs” 1202-3). Examples of time-basedautomation criteria include, but are not limited to, a time of day(e.g., 10:30 AM) and a relative time (e.g., sunrise or sunset).

In some embodiments, the automation criteria are based at least in parton a state of an accessory or a condition or event detected by anaccessory (e.g., by a sensor). Examples of criteria based on a state ofan accessory include, but are not limited to, an accessory beingcontrolled (e.g., “An accessory is controlled” 1202-4) and an accessorybeing triggered or changing state (e.g., an alarm, motion sensitivelight, rain sensor, etc.). Examples of criteria based on a condition orevent detected by an accessory include, but are not limited to: weatherevents, notification events (e.g., receiving a notification), thepresence or absence of a user at a location, the absence of an expectedevent, machine learning, or manual (user) input as described in U.S.patent application Ser. No. 14/725,912 titled “Accessory managementSystems Using Environment Model,” filed May 29, 2015, which isincorporated by reference above.

Returning to FIG. 12A, Device 600 detects a user input corresponding toselection of a first criterion. In the embodiment shown in FIG. 12A, theuser input includes a contact 1220-1 on a representation 1202-1 of afirst criterion labeled “I leave home.” An example of the “I leave home”criterion is described above. In response to detecting user input1220-1, device 600 associates the first criterion with an automationprofile by including the first criterion in the automation criteria ofthe automation profile. As mentioned above, in addition to theautomation criteria, the automation profile also includes datarepresenting at least one designated state for at least one accessorysituated at the location.

FIG. 12A also shows user input 1220-2 on an affordance labeled “Controlan accessory.” In response to detecting user input 1220-2, device 600displays one or more representations of accessories at the location. Insome embodiments, device 600 displays a user interface analogous to userinterface 910-3 described above with reference to FIG. 9C. In FIG. 9C,representations of accessories at the location are displayed and can beselected to be included in a scene profile. Designated states of theaccessories can also be selected for the scene profile. Analogous userinterfaces and techniques can be used to select an accessory and acorresponding designated state of the accessory in order to add dataindicating the designated state of an accessory to the automationprofile. Similar to FIG. 9C, in some embodiments, the representations ofthe accessories include an indication of the current states of theaccessories. Also, similar to the predefined scenes described above withreference to FIGS. 9A and 9F, in some embodiments, the representationsof the accessories include an indication of suggested states of theaccessories. In some embodiments, the designated state is the currentstate of the accessory. In some embodiments, the designated state is astate selected by the user by one of the techniques described above withreference to FIGS. 6B-6K.

In the embodiment illustrated in FIG. 12A, a user can also choose to runa scene profile as part of the automation. In response to user input1220-3 corresponding to selection of affordance 1204-1, device 600displays a list of any scene profiles associated with the location, asshown for example in FIG. 12B, where the scene profiles include dataregarding designated states of accessories associated with the location.In the embodiment illustrated in FIG. 12B, the scene profiles areorganized by room. In some embodiments, a scene profile is listed undera room if it includes an accessory associated with that room. A user canselect one or more of the listed scene profiles to add data indicatingthe designated states of the accessories in the scene profile to theautomation profile.

In some embodiments, after adding data indicating the designated stateof an accessory to the automation profile, device 600 displays a summaryof the automation profile based on the first accessory. An exemplarysummary of an automation profile is illustrated by element 634 in FIG.6Q.

After the automation criteria and accessory data is established for theautomation profile, device 600 determines whether the automationcriteria have been met, and in accordance with a determination that theautomation criteria have been met, sends instructions to set the firstaccessory to the designated state. In some embodiments, device 600 sendsthe instructions directly to the accessory. In some embodiments, device600 sends the instructions to a controller, server, host, or otherexternal device that is configured to control the accessory.

In some embodiments, device 600 sends the instructions automaticallyonce it determines that the automation criteria have been met. Forexample, device 600 sends the instructions automatically if option1206-1 labeled “Automatically” in FIG. 12A is selected. In someembodiments, in accordance with the determination that the automationcriteria are met, device 600 displays a notification or request thatidentifies the automation profile and provides options (e.g.,affordances) to confirm or deny implementation of the automationprofile. For example, a request to confirm implementation of theautomation profile is displayed if option 1206-2 labeled “Always ask” inFIG. 12A is selected. FIG. 12C illustrates an exemplary embodiment of anotification 1208 including a confirm affordance 1212 and a cancelaffordance 1214. In accordance with user input corresponding toconfirmation of implementation of the automation profile (e.g., a tap onaffordance 1212), device 600 sends instructions to implement thedesignated states of the automation profile. In accordance with userinput corresponding to non-confirmation or denial of implementation ofthe automation profile (e.g., a tap on affordance 1214), device 600forgoes sending instructions to implement the designated states of theautomation profile or sends instructions not to implement the designatedstates of the automation profile.

FIG. 13 is a flow diagram illustrating a method for managingcontrollable external devices using an electronic device in accordancewith some embodiments. Method 1300 is performed at a device (e.g., 100,300, 500, or 600) with a display and, optionally, a touch-sensitivesurface. Some operations in method 1300 are, optionally, combined, theorder of some operations is, optionally, changed, and some operationsare, optionally, omitted.

As described below, method 1300 provides an intuitive way for managingcontrollable external devices. The method reduces the cognitive burdenon a user for managing controllable external devices, thereby creating amore efficient human-machine interface. For battery-operated computingdevices, enabling a user to manage controllable external devices fasterand more efficiently conserves power and increases the time betweenbattery charges.

At block 1302, the device detects a first user input (e.g., 1220-1)corresponding to selection of a first criterion (e.g., 1202-1).

At block 1304, in response to detecting the first user input, the deviceassociates the first criterion with an automation profile, where theautomation profile includes data representing: automation criteriaincluding at least the first criterion, and at least one designatedstate for at least one controllable external device situated at alocation. Optionally, the automation criteria are based at least in parton the position of the electronic device relative to the location.Optionally, the automation criteria are based at least in part on atime. Optionally, the automation criteria are based at least in part ona state of a controllable external device at the location.

At block 1306, the device detects a second user input corresponding toselection of a first controllable external device situated at thelocation (e.g., 920-2). Optionally, the selection of the firstcontrollable external device includes selection of a scene profile(e.g., FIG. 12B), and the scene profile includes data regardingdesignated states of a plurality of controllable external devices at thelocation including the first controllable external device.

At block 1308, the device adds data indicating a designated state of thefirst controllable external device to the automation profile.

Optionally, at block 1310, the device displays a summary of theautomation profile based on the first controllable external device(e.g., FIG. 6Q).

At block 1312, the device determines whether the automation criteriahave been met.

At block 1314, in accordance with a determination that the automationcriteria have been met, the device sends instructions to set the firstcontrollable external device to the designated state.

Optionally, at block 1316, the device: detects a third user inputcorresponding to selection of a second controllable external devicesituated at the location; determines whether the third user input meetsinput threshold criteria; in accordance with a determination that thethird user input meets the input threshold criteria, displays anindication of a first state of the second controllable external device(e.g., FIG. 6D, 608); detects a fourth user input corresponding toselection of the first state of the second controllable external device;and in response to detecting the fourth user input, includes dataidentifying the second controllable external device and the first stateof the second controllable external device in the automation profile.

Note that details of the processes described above with respect tomethod 1300 (e.g., FIG. 13) are also applicable in an analogous mannerto the methods described above and below. For example, methods 700,1000, 1600, 1900, 2100, 2300, 2600, 2900, and 3200 optionally includeone or more of the characteristics of the various methods describedabove with reference to method 1300.

In accordance with some embodiments, FIG. 14 shows an exemplaryfunctional block diagram of an electronic device 1400 configured inaccordance with the principles of the various described embodiments. Inaccordance with some embodiments, the functional blocks of electronicdevice 800 are configured to perform the techniques described above. Thefunctional blocks of the device 1400 are, optionally, implemented byhardware, software, or a combination of hardware and software to carryout the principles of the various described examples. It is understoodby persons of skill in the art that the functional blocks described inFIG. 14 are, optionally, combined or separated into sub-blocks toimplement the principles of the various described examples. Therefore,the description herein optionally supports any possible combination orseparation or further definition of the functional blocks describedherein.

As shown in FIG. 14, an electronic device 1400 includes a display unit1402 configured to display a graphic user interface, optionally, atouch-sensitive surface unit 1404 configured to receive contacts, and aprocessing unit 1406 coupled to the display unit 1402 and, optionally,the touch-sensitive surface unit 1404. In some embodiments, theprocessing unit 1406 includes a detecting unit 1408, an associating unit1410, an adding unit 1412, a determining unit 1414, a causing unit 1416,a display enabling unit 1418, and an including unit 1420.

The processing unit 1406 is configured to: detect (e.g., with detectingunit 1408) a first user input corresponding to selection of a firstcriterion; in response to detecting the first user input, associate(e.g., with associating unit 1410) the first criterion with anautomation profile, where the automation profile includes datarepresenting: automation criteria including at least the firstcriterion, and at least one designated state for at least onecontrollable external device situated at a location; detect (e.g., withdetecting unit 1408) a second user input corresponding to selection of afirst controllable external device situated at the location; add (e.g.,with adding unit 1412) data indicating a designated state of the firstcontrollable external device to the automation profile; determine (e.g.,with determining unit 1414) whether the automation criteria have beenmet; and in accordance with a determination that the automation criteriahave been met, cause (e.g., with causing unit 1416) sending ofinstructions to set the first controllable external device to thedesignated state.

In some embodiments, the selection of the first controllable externaldevice includes selection of a scene profile, and the scene profileincludes data regarding designated states of a plurality of controllableexternal devices at the location including the first controllableexternal device.

In some embodiments, the automation criteria are based at least in parton the position of the electronic device relative to the location. Insome embodiments, the automation criteria are based at least in part ona time. In some embodiments, the automation criteria are based at leastin part on a state of a controllable external device at the location.

In some embodiments, the processing unit is further configured to: afteradding the data indicating the designated state of the firstcontrollable external device to the automation profile, enable display(e.g., with display enabling unit 1418) of a summary of the automationprofile based on the first controllable external device.

In some embodiments, the processing unit is further configured to:detect (e.g., with detecting unit 1408) a third user input correspondingto selection of a second controllable external device situated at thelocation; determine (e.g., with determining unit 1414) whether the thirduser input meets input threshold criteria; in accordance with adetermination that the third user input meets the input thresholdcriteria, enable display (e.g., with display enabling unit 1418) of anindication of a first state of the second controllable external device;detect (e.g., with detecting unit 1408) a fourth user inputcorresponding to selection of the first state of the second controllableexternal device; and in response to detecting the fourth user input,include (e.g., with including unit 1420) data identifying the secondcontrollable external device and the first state of the secondcontrollable external device in the automation profile.

The operations described above with reference to FIG. 13 are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.14. For example, detecting operations 1302, 1306, and 1316, associatingoperation 1304, adding operation 1308, displaying operations 1310 and1316, determining operation 1312 and 1316, sending operation 1314, andincluding operation 1316 are, optionally, implemented by event sorter170, event recognizer 180, and event handler 190. Event monitor 171 inevent sorter 170 detects a contact on touch-sensitive display 112, andevent dispatcher module 174 delivers the event information toapplication 136-1. A respective event recognizer 180 of application136-1 compares the event information to respective event definitions186, and determines whether a first contact at a first location on thetouch-sensitive surface corresponds to a predefined event or sub event,such as activation of an affordance on a user interface. When arespective predefined event or sub-event is detected, event recognizer180 activates an event handler 190 associated with the detection of theevent or sub-event. Event handler 190 optionally utilizes or calls dataupdater 176 or object updater 177 to update the application internalstate 192. In some embodiments, event handler 190 accesses a respectiveGUI updater 178 to update what is displayed by the application.Similarly, it would be clear to a person having ordinary skill in theart how other processes can be implemented based on the componentsdepicted in FIGS. 1A-1B.

FIGS. 15A-15B illustrate exemplary user interfaces for associating anaccessory with a location, in accordance with some embodiments. The userinterfaces in these figures are used to illustrate the processesdescribed below, including the processes in FIG. 16.

In some embodiments, an accessory is associated with a location (e.g.,added to a location profile) using an image (e.g., a barcode, quickresponse (QR) code, or the like) that identifies the accessory. Theimage may be included, for example, on the accessory itself or materialsprovided or associated with the accessory (e.g., packaging,instructions, etc.).

FIG. 15A illustrates device 600 displaying a user interface screen1510-1 including instructions to capture an image of a pattern (e.g.,“Scan Code”). Device 600 displays a live video feed from a cameraassociated with device 600 in area 1502 of user interface screen 1510-1to help the user align the image with the field of view of the camera.In some embodiments, user interface screen 1510-1 is displayed inresponse to selection of the affordance shown in FIG. 6M labeled “AddAccessory”.

Using the camera, device 600 captures an image of pattern 1504. Device600 then determines that the pattern corresponds to an accessory. Insome embodiments, instead of capturing the image itself, device 600obtains the image from an external device. In some embodiments, device600 compares the pattern in the image to accessory information stored ina data repository (e.g., a local or remote memory, database, etc.). Insome embodiments, device 600 also receives identification data from theaccessory (e.g., via Bluetooth communication) and compares it with thepattern in the image to determine whether the pattern corresponds to theaccessory. In this way, device 600 verifies that it is correctlyidentifying the accessory associated with the image.

In response to a determination that the pattern corresponds to theaccessory, device 600 associates the accessory with a location. In someembodiments, device 600 creates a profile (an “accessory profile”)associated with the accessory, where the profile includes datarepresenting the operational states of the accessory. In someembodiments, in response to determining that the pattern corresponds toan accessory, device 600 establishes a paired relationship with theaccessory which enables the electronic device to exchange informationwith the accessory (e.g., via fear-field communication, Bluetoothprotocol, Internet, a local network, etc.) so that device 600 canmonitor and/or control the state of the accessory.

In some embodiments, device 600 displays a user interface for inputtinginformation about the accessory, including assigning the accessory to aroom or designated area of the location. FIG. 15B illustrates a userinterface screen 1510-2 that displays information about the accessory, arepresentation 1506 of the accessory, and options 1508-1 and 1508-2 forinputting information about the accessory. User interface screen 1510-2and/or representation 1506 can be populated, e.g., automatically basedon information obtained from capturing the image and identifying theaccessory. In some embodiments, information is obtained manually basedon input from a user. In the embodiment illustrated in FIG. 15B, therepresentation of the accessory includes an icon, a name, and a roomassociated with the accessory. In some embodiments, the designated name,image, room, or other information is included in a data profile of theaccessory.

To associate the accessory with a room, user interface screen 1510-2includes an affordance 1508-2 labeled “Choose Room” that when selectedallows a user to designate a room. In some embodiments, the devicedisplays a list of rooms, drop-down menu, text entry field, etc. for auser to select a room. In some embodiments, a list of rooms is displayedand arranged based on the position of device 600 in relation to thelocation. For example, in some embodiments, device 600 estimates whichroom it is in based on its position and presents the estimated room as afirst option. Device 600 detects user input representing selection of aroom (e.g., a tap on a room in a dropdown menu), and in response todetecting the user input, associates the selected room with theaccessory. In some embodiments, a similar technique is used to input aname for the accessory. In some embodiments, device 600 updates userinterface screen 1510-2 and/or representation 1506 to include the roominformation in the profile associated with the accessory.

User interface screen 1510-2 also includes icons 1512 with images thatcan be associated with the accessory in response to selection of thecorresponding icon. User interface screen 1510-2 also includes anaffordance 1514 for designating the accessory as a “Favorite.” In someembodiments, device 600 detects a user input on affordance 1541, and inresponse, displays a representation of the accessory on the Home screen610-1, e.g., under the Favorite Accessories heading.

FIG. 16 is a flow diagram illustrating a method for managingcontrollable external devices using an electronic device in accordancewith some embodiments. Method 1600 is performed at a device (e.g., 100,300, 500, or 600) with a display, optionally, a touch-sensitive surface,and, optionally, a camera. Some operations in method 1600 are,optionally, combined, the order of some operations is, optionally,changed, and some operations are, optionally, omitted.

As described below, method 1600 provides an intuitive way for managingcontrollable external devices. The method reduces the cognitive burdenon a user for managing controllable external devices, thereby creating amore efficient human-machine interface. For battery-operated computingdevices, enabling a user to manage controllable external devices fasterand more efficiently conserves power and increases the time betweenbattery charges.

Optionally, at block 1602, the device displays a first user interfacescreen (e.g., 1510-1)) including instructions to capture an image of apattern. Optionally, the first user interface screen is displayed inresponse to selection of a first affordance on a second user interfacescreen (e.g., FIG. 6M).

At block 1604, the device captures, using the camera, the image of thepattern (e.g., 1504).

At block 1606, the device determines that the pattern corresponds to acontrollable external device, where the controllable external device isconfigured to operate in two or more states and is remotely controllablebetween the two or more states.

At block 1608, in response to a determination that the patterncorresponds to the controllable external device, the device associatesthe controllable external device with a location, where the location hasat least one designated room.

Optionally, at block 1610, after determining that the patterncorresponds to a controllable external device, the device displays arepresentation of the controllable external device (e.g., 1506).

At block 1612, the device detects a first user input representingselection of a room from the at least one designated room (e.g.,1508-2).

At block 1614, in response to detecting the first user input, the deviceassociates the selected room with the controllable external device.

Optionally, at block 1616, the device detects a second user input (e.g.,selection of affordance 1514) after determining that the patterncorresponds to a controllable external device; and in response todetecting the second user input, displays a representation of thecontrollable external device on the first user interface screen (e.g.,610-1).

Note that details of the processes described above with respect tomethod 1600 (e.g., FIG. 16) are also applicable in an analogous mannerto the methods described above and below. For example, methods 700,1000, 1300, 1900, 2100, 2300, 2600, 2900, and 3200 optionally includeone or more of the characteristics of the various methods describedabove with reference to method 1600.

In accordance with some embodiments, FIG. 17 shows an exemplaryfunctional block diagram of an electronic device 1700 configured inaccordance with the principles of the various described embodiments. Inaccordance with some embodiments, the functional blocks of electronicdevice 1700 are configured to perform the techniques described above.The functional blocks of the device 1700 are, optionally, implemented byhardware, software, or a combination of hardware and software to carryout the principles of the various described examples. It is understoodby persons of skill in the art that the functional blocks described inFIG. 17 are, optionally, combined or separated into sub-blocks toimplement the principles of the various described examples. Therefore,the description herein optionally supports any possible combination orseparation or further definition of the functional blocks describedherein.

As shown in FIG. 17, an electronic device 1700 includes a display unit1702 configured to display a graphic user interface, optionally, atouch-sensitive surface unit 1704 configured to receive contacts,optionally, a camera unit 1706 configured to capture images, and aprocessing unit 1708 coupled to the display unit 1702, optionally, thetouch-sensitive surface unit 1704, and, optionally, the camera unit1706. In some embodiments, the processing unit 1708 includes a causingunit 1710, a determining unit 1712, an associating unit 1714, adetecting unit 1716, and a display enabling unit 1718.

The processing unit 1708 is configured to: cause (e.g., causing unit1710) capturing, using the camera unit, an image of a pattern; determine(e.g., with determining unit 1712) that the pattern corresponds to acontrollable external device, where the controllable external device isconfigured to operate in two or more states and is remotely controllablebetween the two or more states; in response to a determination that thepattern corresponds to the controllable external device, associate(e.g., with associating unit 1714) the controllable external device witha location, where the location has at least one designated room; detect(e.g., with detecting unit 1716) a first user input representingselection of a room from the at least one designated room; and inresponse to detecting the first user input, associate (e.g., with theassociating unit 1714) the selected room with the controllable externaldevice.

In some embodiments, the processing unit 1708 is further configured to:before the image of the pattern is captured, enable display (e.g., withdisplay enabling unit 1718) of a first user interface screen includinginstructions to capture the image of the pattern. In some embodiments,the first user interface screen is displayed in response to selection ofa first affordance on a second user interface screen. In someembodiments, the processing unit is further configured to: afterdetermining that the pattern corresponds to a controllable externaldevice, detect (e.g., with detecting unit 1716) a second user input; andin response to detecting the second user input, enable display (e.g.,with display enabling unit 1718) of a representation of the controllableexternal device on the second user interface screen.

In some embodiments, the processing unit 1708 is further configured to:after determining that the pattern corresponds to a controllableexternal device, enable display (e.g., with display enabling unit 1718)of a representation of the controllable external device.

The operations described above with reference to FIG. 16 are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.17. For example, displaying operations 1602 and 1610, capturingoperation 1604, determining operation 1606, associating operations 1608and 1614, and detecting operations 1612 and 1616 are, optionally,implemented by event sorter 170, event recognizer 180, and event handler190. Event monitor 171 in event sorter 170 detects a contact ontouch-sensitive display 112, and event dispatcher module 174 deliversthe event information to application 136-1. A respective eventrecognizer 180 of application 136-1 compares the event information torespective event definitions 186, and determines whether a first contactat a first location on the touch-sensitive surface corresponds to apredefined event or sub event, such as activation of an affordance on auser interface. When a respective predefined event or sub-event isdetected, event recognizer 180 activates an event handler 190 associatedwith the detection of the event or sub-event. Event handler 190optionally utilizes or calls data updater 176 or object updater 177 toupdate the application internal state 192. In some embodiments, eventhandler 190 accesses a respective GUI updater 178 to update what isdisplayed by the application. Similarly, it would be clear to a personhaving ordinary skill in the art how other processes can be implementedbased on the components depicted in FIGS. 1A-1B.

FIGS. 18A-18C illustrate exemplary user interfaces for setting criteriafor an automation profile associated with a location, in accordance withsome embodiments. The user interfaces in these figures are used toillustrate the processes described below, including the processes inFIG. 19.

FIG. 18A illustrates user interface screen 1810-1 displaying a map view1802. Map view 1802 is substantially centered on the location associatedwith the automation profile. Map view 1802 also includes an indication1804 of a location-based criteria, and in particular, a circularthreshold around the position of the location. The location-basedcriteria are based at least in part on the position of device 600 withrespect to the location. In some embodiments, the location-basedcriteria include whether device 600 is within a threshold distance ofthe location, where the threshold distance is represented by the radiusof the circle. In some embodiments, the threshold distance is the samein all directions from the location (e.g., a circular boundary). In someembodiments, the threshold distance varies with direction from thelocation.

In some embodiments, the location-based criteria include whether device600 is less than a threshold distance from the location. In someembodiments, the location-based criteria include whether device 600 isgreater than a threshold distance from the location. In someembodiments, the location-based criteria include whether device 600 hasmoved from greater than a threshold distance from the location to lessthan the threshold distance from the location. This criterion optionallycorresponds to the “I arrive home” criteria 1202-2 represented in FIG.12A. In some embodiments, the location-based criteria include whetherdevice 600 has moved from less than a threshold distance from the firstlocation to greater than the threshold distance from the location. Thiscriterion optionally corresponds to the “I leave home” criteria 1202-1represented in FIG. 12A.

Device 600 detects user input corresponding to designation of thelocation-based criteria (e.g., selection of criteria 1202-1 displayed inFIG. 12A). In response to detecting the user input corresponding todesignation of the criteria, device 600 adds the location-based criteriato the automation criteria of the automation profile associated with thelocation.

User interface screen 1810-1 also includes additional selectablecriteria or triggering conditions for the automation profile. Userinterface objects 1804-1 to 1804-4 represent respective time-basedcriteria that determine when an automation profile can be enabled: “Allday”, “Only if it's daytime”, “Only if it is night”, and a specifiedwindow of time (e.g., “From” and “To”). User interface screen 1810-1also includes options 1806-1 to 1806-7 for selecting on what days, ifany, an automation is repeated.

Device 600 also provides an option to further specify the time-basedcriteria of selected objects 1804-1 to 1804-4. In some embodiments, inresponse to selection of info icon 1808 on user interface screen 1810-1,device 600 displays user interface screen 1810-2 illustrated in FIG.18B, which provides additional time-based criteria for the automationprofile including: relative start and end times (e.g., sunrise andsunset), specific start time (e.g., time of day), specific ending time,relative ending time, duration, and time-based constraints (e.g., onlyon Monday). Device 600 also optionally provides the option to specifythe time-based criteria even further for a selected relative start andend time. In some embodiments, in response to selection of info icon1812 on user interface screen 1810-2, device 600 displays user interfacescreen 1810-3 illustrated in FIG. 18C, which provides the option to addor subtract an offset from a relative time (e.g., sunrise).

Device 600 detects user input corresponding to designation of thetime-based criteria (e.g., a tap on one or more of user interfaceobjects shown in FIGS. 18A-18C), and in response, adds the selectedtime-based criteria to the automation criteria of the automation profilefor controlling the at least one affordance associated with theautomation profile.

In some embodiments, the user interfaces and time-based criteriadescribed with respect to FIGS. 18A-18C are provided during the creationand/or configuration of an automation profile as described above withreference to FIGS. 12A-12B (e.g., “I leave home” 1202-1 and “I arrivehome” 1202-2). Conversely, in some embodiments, the automation criteriaare based in part on a state of an accessory at the location asdescribed with reference to the event criteria 1202-4, “An accessory iscontrolled”, in FIG. 12A. (e.g., automation criteria is based on whetheran accessory is controlled and/or triggered or an accessory detects anevent).

Once the automation criteria are established, the position of device 600is determined. In some embodiments, device 600 determines its position,e.g., via GPS, WiFi, or other signal. In some embodiments, device 600determines its position based on information obtained from an externaldevice.

Device 600 determines whether the automation criteria are met based atleast in part on the determined position. In accordance with adetermination that the automation criteria is met, device 600 sendsinstructions to implement the automation profile, which includes settingat least one accessory to a designated state. In some embodiments, inaccordance with a determination that the automation criteria are notmet, device 600 forgoes sending instructions to implement the automationprofile.

FIG. 19 is a flow diagram illustrating a method for managingcontrollable external devices using an electronic device in accordancewith some embodiments. Method 1900 is performed at a device (e.g., 100,300, 500, or 600) with a display and, optionally, a touch-sensitivesurface. Some operations in method 1900 are, optionally, combined, theorder of some operations is, optionally, changed, and some operationsare, optionally, omitted.

As described below, method 1900 provides an intuitive way for managingcontrollable external devices. The method reduces the cognitive burdenon a user for managing controllable external devices, thereby creating amore efficient human-machine interface. For battery-operated computingdevices, enabling a user to manage controllable external devices fasterand more efficiently conserves power and increases the time betweenbattery charges.

At block 1902, the device detects user input (e.g., 1220-1)corresponding to designation of location-based criteria (e.g., 1202-1),where the location-based criteria are based at least in part on thelocation of the electronic device with respect to a first locationhaving at least one controllable external device. Optionally, the devicedisplays a map view (e.g., 1802) with an indication (e.g., 1804) of thelocation-based criteria. Optionally, the location-based criteria includeone or more of: whether the electronic device is less than a thresholddistance from the first location, whether the electronic device isgreater than a threshold distance from the first location, whether theelectronic device has moved from greater than a threshold distance fromthe first location to less than the threshold distance from thelocation, and whether the electronic device has moved from less than athreshold distance from the first location to greater than the thresholddistance from the location.

At block 1904, in response to detecting the user input corresponding todesignation of the criteria, the device adds the location-based criteriato an automation profile associated with the location, where theautomation profile includes data representing automation criteria forcontrolling the at least one controllable external device and adesignated state of the at least one controllable external device, andthe automation criteria for controlling the at least one controllableexternal device includes the location-based criteria. Optionally, theautomation criteria for controlling the at least one controllableexternal device are based in part on a state of a first controllableexternal device of the at least one controllable external device (e.g.,1202-4).

At block 1906, the device determines the location of the electronicdevice.

At block 1908, the device determines whether the automation criteria forcontrolling the at least one controllable external device is met basedat least in part on the determined location of the electronic device.

At block 1910, in accordance with a determination that the automationcriteria for controlling the at least one controllable external deviceis met, the device sends instructions to set the at least onecontrollable external device to the designated state.

Optionally, at block 1912, in accordance with a determination that theautomation criteria for controlling the at least one controllableexternal device is not met, the device forgoes sending instructions toset the at least one controllable external device to the designatedstate.

Optionally, at block 1914, the device detects user input correspondingto designation of a time-based criteria (e.g., 1202-3 and 1804-1 to1804-4); and in response to detecting the user input, adds thetime-based criteria to the automation criteria for controlling the atleast one controllable external device.

Note that details of the processes described above with respect tomethod 1900 (e.g., FIG. 19) are also applicable in an analogous mannerto the methods described above and below. For example, methods 700,1000, 1300, 1600, 2100, 2300, 2600, 2900, and 3200 optionally includeone or more of the characteristics of the various methods describedabove with reference to method 1900.

In accordance with some embodiments, FIG. 20 shows an exemplaryfunctional block diagram of an electronic device 2000 configured inaccordance with the principles of the various described embodiments. Inaccordance with some embodiments, the functional blocks of electronicdevice 2000 are configured to perform the techniques described above.The functional blocks of the device 2000 are, optionally, implemented byhardware, software, or a combination of hardware and software to carryout the principles of the various described examples. It is understoodby persons of skill in the art that the functional blocks described inFIG. 20 are, optionally, combined or separated into sub-blocks toimplement the principles of the various described examples. Therefore,the description herein optionally supports any possible combination orseparation or further definition of the functional blocks describedherein.

As shown in FIG. 20, an electronic device 2000 includes a display unit2002 configured to display a graphic user interface, optionally, atouch-sensitive surface unit 2004 configured to receive contacts, and aprocessing unit 2006 coupled to the display unit 2002 and, optionally,the touch-sensitive surface unit 2004. In some embodiments, theprocessing unit 2006 includes a detecting unit 2008, an adding unit2010, a determining unit 2012, a causing unit 2014, and a displayenabling unit 2016.

The processing unit 2006 is configured to: detect (e.g., with detectingunit 2008) user input corresponding to designation of location-basedcriteria, where the location-based criteria are based at least in parton the location of the electronic device with respect to a firstlocation having at least one controllable external device; in responseto detecting the user input corresponding to designation of thecriteria, add (e.g., with adding unit 2010) the location-based criteriato an automation profile associated with the location, where theautomation profile includes data representing automation criteria forcontrolling the at least one controllable external device and adesignated state of the at least one controllable external device, andwhere the automation criteria for controlling the at least onecontrollable external device includes the location-based criteria;determine (e.g., with determining unit 2012) the location of theelectronic device; determine (e.g., with determining unit 2012) whetherthe automation criteria for controlling the at least one controllableexternal device is met based at least in part on the determined locationof the electronic device; and in accordance with a determination thatthe automation criteria for controlling the at least one controllableexternal device is met, cause (e.g., with causing unit 2014) sending ofinstructions to set the at least one controllable external device to thedesignated state.

In some embodiments, the processing unit 2006 is further configured to:in accordance with a determination that the automation criteria forcontrolling the at least one controllable external device is not met,forgoing sending instructions to set the at least one controllableexternal device to the designated state.

In some embodiments, the processing unit 2006 is further configured toenable display (e.g., with display enabling unit 2016) of a map viewwith an indication of the location-based criteria.

In some embodiments, the location-based criteria include whether theelectronic device is less than a threshold distance from the firstlocation. In some embodiments, the location-based criteria includewhether the electronic device is greater than a threshold distance fromthe first location. In some embodiments, the location-based criteriainclude whether the electronic device has moved from greater than athreshold distance from the first location to less than the thresholddistance from the location. In some embodiments, the location-basedcriteria include whether the electronic device has moved from less thana threshold distance from the first location to greater than thethreshold distance from the location.

In some embodiments, the processing unit 2006 is further configured to:detect (e.g., with detecting unit 2008) user input corresponding todesignation of a time-based criteria; and in response to detecting theuser input, add (e.g., with adding unit 2010) the time-based criteria tothe automation criteria for controlling the at least one controllableexternal device.

In some embodiments, the automation criteria for controlling the atleast one controllable external device are based in part on a state of afirst controllable external device of the at least one controllableexternal device.

The operations described above with reference to FIG. 19 are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.20. For example, detecting operations 1902 and 1914, adding operations1904 and 1914, determining operations 1906 and 1908, sending operation1910, and forgoing operation 1912 are, optionally, implemented by eventsorter 170, event recognizer 180, and event handler 190. Event monitor171 in event sorter 170 detects a contact on touch-sensitive display112, and event dispatcher module 174 delivers the event information toapplication 136-1. A respective event recognizer 180 of application136-1 compares the event information to respective event definitions186, and determines whether a first contact at a first location on thetouch-sensitive surface corresponds to a predefined event or sub event,such as activation of an affordance on a user interface. When arespective predefined event or sub-event is detected, event recognizer180 activates an event handler 190 associated with the detection of theevent or sub-event. Event handler 190 optionally utilizes or calls dataupdater 176 or object updater 177 to update the application internalstate 192. In some embodiments, event handler 190 accesses a respectiveGUI updater 178 to update what is displayed by the application.Similarly, it would be clear to a person having ordinary skill in theart how other processes can be implemented based on the componentsdepicted in FIGS. 1A-1B.

FIG. 21 is a flow diagram illustrating a method for managingcontrollable external devices using an electronic device in accordancewith some embodiments. Method 2100 is performed at a device (e.g., 100,300, 500, or 600) with a display and, optionally, a touch-sensitivesurface. Some operations in method 2100 are, optionally, combined, theorder of some operations is, optionally, changed, and some operationsare, optionally, omitted.

As described below, method 2100 provides an intuitive way for managingcontrollable external devices. The method reduces the cognitive burdenon a user for managing controllable external devices, thereby creating amore efficient human-machine interface. For battery-operated computingdevices, enabling a user to manage controllable external devices fasterand more efficiently conserves power and increases the time betweenbattery charges.

At block 2102, the device determines whether criteria of an automationprofile (e.g., 1202-1 to 1202-4 and 1804-1 to 1804-4) are met, where theautomation profile includes data representing criteria for controlling astate of at least one controllable external device and designated statesfor the at least one controllable external device. Optionally, thecriteria of the automation profile are based at least in part on one ormore of: the location of the electronic device with respect to alocation associated with the at least one controllable external device,time, and a state of a first controllable external device of the atleast one controllable external device (e.g., FIGS. 12A and 18A).

At block 2104, in accordance with a determination that the criteria ofthe automation profile are met, the device displays a notification(e.g., 1208), where the notification includes an indication thatidentifies the automation profile and an indication to confirmimplementation of the automation profile (e.g., 1212 in FIG. 12C).

At block 2106, the device detects a user input (e.g., selection ofaffordance 1212 or 1214).

At block 2108, in response to detecting the user input, the device: inaccordance with the user input corresponding to confirmation ofimplementation of the automation profile (e.g., selection of affordance1212), sends instructions to implement the designated states of theautomation profile; and in accordance with the user input correspondingto non-confirmation of implementation of the automation profile (e.g.,selection of affordance 1214), sends instructions not to implement thedesignated states of the automation profile.

Note that details of the processes described above with respect tomethod 2100 (e.g., FIG. 21) are also applicable in an analogous mannerto the methods described above and below. For example, methods 700,1000, 1300, 1600, 1900, 2300, 2600, 2900, and 3200 optionally includeone or more of the characteristics of the various methods describedabove with reference to method 2100.

In accordance with some embodiments, FIG. 22 shows an exemplaryfunctional block diagram of an electronic device 2200 configured inaccordance with the principles of the various described embodiments. Inaccordance with some embodiments, the functional blocks of electronicdevice 2200 are configured to perform the techniques described above.The functional blocks of the device 2200 are, optionally, implemented byhardware, software, or a combination of hardware and software to carryout the principles of the various described examples. It is understoodby persons of skill in the art that the functional blocks described inFIG. 22 are, optionally, combined or separated into sub-blocks toimplement the principles of the various described examples. Therefore,the description herein optionally supports any possible combination orseparation or further definition of the functional blocks describedherein.

As shown in FIG. 22, an electronic device 2200 includes a display unit2202 configured to display a graphic user interface, optionally, atouch-sensitive surface unit 2204 configured to receive contacts, and aprocessing unit 2206 coupled to the display unit 2202 and, optionally,the touch-sensitive surface unit 2204. In some embodiments, theprocessing unit 2206 includes a determining unit 2208, a displayenabling unit 2210, a detecting unit 2212, and a causing unit 2214.

The processing unit 2206 is configured to: determine (e.g., withdetermining unit 2208) whether criteria of an automation profile aremet, where the automation profile includes data representing criteriafor controlling a state of at least one controllable external device anddesignated states for the at least one controllable external device; inaccordance with a determination that the criteria of the automationprofile are met, enable display (e.g., with display enabling unit 2210)of a notification, where the notification includes an indication thatidentifies the automation profile and an indication to confirmimplementation of the automation profile; detect (e.g., with detectingunit 2212) a user input; and in response to detecting the user input: inaccordance with the user input corresponding to confirmation ofimplementation of the automation profile, cause (e.g., with causing unit2214) sending of instructions to implement the designated states of theautomation profile; and in accordance with the user input correspondingto non-confirmation of implementation of the automation profile, cause(e.g., with causing unit 2214) sending of instructions not to implementthe designated states of the automation profile.

In some embodiments, the criteria of the automation profile are based atleast in part on the location of the electronic device with respect to alocation associated with the at least one controllable external device.In some embodiments, the criteria of the automation profile are based atleast in part on time. In some embodiments, the criteria of theautomation profile are based at least in part on a state of a firstcontrollable external device of the at least one controllable externaldevice.

The operations described above with reference to FIG. 21 are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.22. For example, determining operation 2102, displaying operation 2104,detecting operation 2106 and sending operation 2108 are, optionally,implemented by event sorter 170, event recognizer 180, and event handler190. Event monitor 171 in event sorter 170 detects a contact ontouch-sensitive display 112, and event dispatcher module 174 deliversthe event information to application 136-1. A respective eventrecognizer 180 of application 136-1 compares the event information torespective event definitions 186, and determines whether a first contactat a first location on the touch-sensitive surface corresponds to apredefined event or sub event, such as activation of an affordance on auser interface. When a respective predefined event or sub-event isdetected, event recognizer 180 activates an event handler 190 associatedwith the detection of the event or sub-event. Event handler 190optionally utilizes or calls data updater 176 or object updater 177 toupdate the application internal state 192. In some embodiments, eventhandler 190 accesses a respective GUI updater 178 to update what isdisplayed by the application. Similarly, it would be clear to a personhaving ordinary skill in the art how other processes can be implementedbased on the components depicted in FIGS. 1A-1B.

FIG. 23 is a flow diagram illustrating a method for managingcontrollable external devices using an electronic device in accordancewith some embodiments. Method 2300 is performed at a device (e.g., 100,300, 500, or 600) with a display and, optionally, a touch-sensitivesurface. Some operations in method 2300 are, optionally, combined, theorder of some operations is, optionally, changed, and some operationsare, optionally, omitted.

As described below, method 2300 provides an intuitive way for managingcontrollable external devices. The method reduces the cognitive burdenon a user for managing controllable external devices, thereby creating amore efficient human-machine interface. For battery-operated computingdevices, enabling a user to managing controllable external devicesfaster and more efficiently conserves power and increases the timebetween battery charges.

At block 2302, the device displays a representation (e.g., 620-1) of acontrollable external device, where the controllable external device isconfigured to operate in a plurality of states, and where thecontrollable external device is in a first state of a plurality ofstates. Optionally, the representation of the controllable externaldevice includes an indication of the current state of the controllableexternal device. In some embodiments, the states of the plurality ofstates are continuous (e.g., FIGS. 6J-6K). In some embodiments, thestates of the plurality of states are discrete (e.g., FIGS. 6D-6I). Insome embodiments, the controllable external device is a color-adjustablelight bulb, and the plurality of states includes a plurality of colors(e.g., FIG. 6K).

At block 2304, the device detects a first user input corresponding to aselection of the controllable external device (e.g., 630-1).

At block 2306, the device determines whether the first user input meetsinput criteria.

At block 2308, in accordance with a determination that the first userinput does not meet the input criteria, the device sends instructions toset the state of the controllable external device to a second state ofthe plurality of states (e.g., 620-1 in FIGS. 6B-6C).

At block 2310, in accordance with a determination that the first userinput meets the input criteria, the device: displays an affordanceindicating the plurality of states of the controllable external device(e.g., 608); while displaying the affordance indicating the plurality ofstates of the controllable external device, detects a second user input(e.g., 630-2) on the affordance corresponding to selection of a thirdstate of the plurality of states; and in response to detecting thesecond user input, sends instructions to set the state of thecontrollable external device to the third state.

Optionally, at block 2312, the device, while displaying the affordanceindicating the plurality of states of the controllable external device,detects a third user input corresponding to selection of a location onthe display that does not correspond to the affordance (e.g., 612); andin response to detecting the third user input, ceases to display theaffordance.

Note that details of the processes described above with respect tomethod 2300 (e.g., FIG. 23) are also applicable in an analogous mannerto the methods described above and below. For example, methods 700,1000, 1300, 1600, 1900, 2100, 2600, 2900, and 3200 optionally includeone or more of the characteristics of the various methods describedabove with reference to method 2300.

In accordance with some embodiments, FIG. 24 shows an exemplaryfunctional block diagram of an electronic device 2400 configured inaccordance with the principles of the various described embodiments. Inaccordance with some embodiments, the functional blocks of electronicdevice 2400 are configured to perform the techniques described above.The functional blocks of the device 2400 are, optionally, implemented byhardware, software, or a combination of hardware and software to carryout the principles of the various described examples. It is understoodby persons of skill in the art that the functional blocks described inFIG. 24 are, optionally, combined or separated into sub-blocks toimplement the principles of the various described examples. Therefore,the description herein optionally supports any possible combination orseparation or further definition of the functional blocks describedherein.

As shown in FIG. 24, an electronic device 2400 includes a display unit2402 configured to display a graphic user interface, optionally, atouch-sensitive surface unit 2404 configured to receive contacts, and aprocessing unit 2406 coupled to the display unit 2402 and, optionally,the touch-sensitive surface unit 2404. In some embodiments, theprocessing unit 2406 includes a display enabling unit 2408, a detectingunit 2410, a determining unit 2412, and a causing unit 2414.

The processing unit 2406 is configured to: enable display (e.g., withdisplay enabling unit 2408) of a representation of a controllableexternal device, where the controllable external device is configured tooperate in a plurality of states, and where the controllable externaldevice is in a first state of a plurality of states; detect (e.g., withdetecting unit 2410) a first user input corresponding to a selection ofthe controllable external device; determine (e.g., with determining unit2412) whether the first user input meets input criteria; in accordancewith a determination that the first user input does not meet the inputcriteria, cause (e.g., with causing unit 2414) sending of instructionsto set the state of the controllable external device to a second stateof the plurality of states; and in accordance with a determination thatthe first user input meets the input criteria: enable display (e.g.,with display enabling unit 2408) of an affordance indicating theplurality of states of the controllable external device; whiledisplaying the affordance indicating the plurality of states of thecontrollable external device, detect a second user input on theaffordance corresponding to selection of a third state of the pluralityof states; and in response to detecting the second user input, cause(e.g., with causing unit 2414) sending of instructions to set the stateof the controllable external device to the third state.

In some embodiments, the states of the plurality of states arecontinuous. In some embodiments, the states of the plurality of statesare discrete. In some embodiments, the controllable external device is acolor-adjustable light bulb, and the plurality of states includes aplurality of colors. In some embodiments, the representation of thecontrollable external device includes an indication of the current stateof the controllable external device.

In some embodiments, the processing unit 2406 is further configured to:while the affordance indicating the plurality of states of thecontrollable external device is displayed, detect (e.g., with detectingunit 2410) a third user input corresponding to selection of a locationon the display that does not correspond to the affordance; and inresponse to detecting the third user input, cause (e.g., with causingunit 2414) display of the affordance to cease.

The operations described above with reference to FIG. 23 are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.24. For example, displaying operations 2302 and 2310, detectingoperations 2304, 2310, and 2312, determining operation 2306, and sendingoperation 2308 and 2310 are, optionally, implemented by event sorter170, event recognizer 180, and event handler 190. Event monitor 171 inevent sorter 170 detects a contact on touch-sensitive display 112, andevent dispatcher module 174 delivers the event information toapplication 136-1. A respective event recognizer 180 of application136-1 compares the event information to respective event definitions186, and determines whether a first contact at a first location on thetouch-sensitive surface corresponds to a predefined event or sub event,such as activation of an affordance on a user interface. When arespective predefined event or sub-event is detected, event recognizer180 activates an event handler 190 associated with the detection of theevent or sub-event. Event handler 190 optionally utilizes or calls dataupdater 176 or object updater 177 to update the application internalstate 192. In some embodiments, event handler 190 accesses a respectiveGUI updater 178 to update what is displayed by the application.Similarly, it would be clear to a person having ordinary skill in theart how other processes can be implemented based on the componentsdepicted in FIGS. 1A-1B.

FIG. 25 illustrates an exemplary user interface for managingcontrollable external devices, in accordance with some embodiments. Theuser interfaces in these figures are used to illustrate the processesdescribed below, including the processes in FIG. 26.

In some embodiments, device 600 determines its position at a firstposition within a location, where the location includes at least onedesignated room. In some embodiments, the position of device 600 isdetermined using GPS, Bluetooth, and/or WiFi signals. Exemplarytechniques for determining the position of device 600 include: comparingthe strength of a signal (or signals) from a first set of devices withknown positions to the strength of a signal (or signals) from a secondset of devices with known positions; (2) determining the strength of asignal received from a WiFi source with a known position (e.g., thestrength of the signal corresponds to a radius that is used to determinea position based on the layout of the location); (3) detecting Bluetoothsignals (e.g., Bluetooth low energy signals) from a beacon device (e.g.,the iBeacon protocol developed by Apple Inc. of Cupertino, Calif.); (4)dead reckoning (e.g., using a previously determined position andadvancing that position based upon known or estimated speeds overelapsed time and course, as determined, for example, by sensors indevice 600); and GPS.

After determining the first position, device 600 determines that thefirst position corresponds to a first room of the location. In responseto determining that the first position corresponds to the first room,device 600 displaying a representation of an accessory associated withthe first room. In some embodiments, device 600 also displays, based onthe determination of the room, a representation of a scene profile,where the scene profile includes data regarding a designated state of atleast one accessory associated with the determined room.

FIG. 25 illustrates an exemplary user interface screen 2500 displayed inresponse to determining that the position corresponds to the bedroom ofa location. User interface screen includes representations 2502 of threeaccessories associated with the bedroom and representations 2504 ofthree scene profiles that include accessories associated with thebedroom. Each representation 2502 of the accessories indicates thecurrent state of the corresponding accessory (e.g., Set to 72 degrees)or an action that selection of the representation would cause (e.g.,Turn OFF or Turn ON, which in some circumstances implies the currentstate). In some embodiments, the representations 2504 includeaffordances for controlling the state of the corresponding accessoriesas described above with reference to FIGS. 6B-6K. In some embodiments,the representations 2504 of the scene profiles can be selected toimplement the scene profile.

In some embodiments, device 600 periodically determines its position andwhether the position corresponds to a designated room in the location.In some embodiments, device 600 automatically displays representationsof the accessories in a room upon determining that its positioncorresponds to a room (e.g., without being prompted by the user). Insome embodiments, device 600 displays a notification in response todetermining that the position corresponds to the room. The notificationcan be selected by a user to display the accessories associated with thedetermined room. These techniques may allow the user to quickly andeasily control the accessories in a room upon approaching or enteringthe room.

Optionally, while displaying the representation of the accessoryassociated with the first room, device 600 detect a user input, and inresponse to detecting the user input, opens an application forcontrolling accessories at the location. For example, as shown in FIG.25, user interface screen 2500 includes an affordance 2506 to open theapplication provides access to control all of the accessories associatedwith the other rooms in a location. In some embodiments, the applicationlaunched by selection of affordance 2506 is the application or anapplication with user interfaces similar to the application describedwith reference to FIGS. 6A-6S. In this way, device 600 can potentiallyreduce the burden on the user to control an accessory in a differentroom if device 600 determines the room incorrectly.

In some embodiments, device 600 determines its position, determines aroom, and/or displays representation of accessories associated with thedetermined room in response to user input. For example, when choosingaccessories for a scene profile or an automation profile as describedabove with reference to FIGS. 9B-9C and 12B, device 600 optionallydisplays the accessories or scene profiles associated with thedetermined room above accessories and scene profiles in other rooms. Forexample, with respect to FIG. 9C, in some embodiments, if device 600determines that it is positioned in the Bathroom, the Bathroom headingand representations 908-4 and 908-5 appear above the Bedroom heading andrepresentations 908-1 to 908-3.

In some embodiments, after displaying a representation of the accessoryassociated with the first room, device 600 determines its position at asecond position within the location, determines that the second positioncorresponds to a second room of the location, where the second room isdifferent than the first room. In response to determining that thesecond position corresponds to the second room, device 600 ceases todisplay the representation of the accessory associated with the firstroom and displays a representation of an accessory associated with thesecond room.

FIG. 26 is a flow diagram illustrating a method for managingcontrollable external devices using an electronic device in accordancewith some embodiments. Method 2600 is performed at a device (e.g., 100,300, 500, or 600) with a display and, optionally, a touch-sensitivesurface. Some operations in method 2600 are, optionally, combined, theorder of some operations is, optionally, changed, and some operationsare, optionally, omitted.

As described below, method 2600 provides an intuitive way for managingcontrollable external devices. The method reduces the cognitive burdenon a user for managing controllable external devices, thereby creating amore efficient human-machine interface. For battery-operated computingdevices, enabling a user to manage controllable external devices fasterand more efficiently conserves power and increases the time betweenbattery charges.

At block 2602, the device determines a first position of the devicewithin a location, where the location includes at least one designatedroom. Optionally, the position of the device is determined using GPS,Bluetooth, and/or WiFi signals.

At block 2604, the device determines that the first position correspondsto a first room of the at least one designated room.

At block 2606, in response to determining that the first positioncorresponds to the first room, the device displays a representation of afirst controllable external device associated with the first room (e.g.,2502). Optionally, the representation of the first controllable externaldevice indicates a current state of the first controllable externaldevice. Optionally, the device displays, based on the determination ofthe room, a representation of a scene profile, where the scene profileincludes data regarding designated states of a plurality of controllableexternal devices at the location including at least one controllableexternal device associated with the determined room.

Optionally, at block 2608, the device detects a first user input (e.g.,selection of one of representations 2502); and in response to the firstuser input, sends instructions to set the first controllable externaldevice to a first state.

Optionally, at block 2610, the device, while displaying therepresentation of the first controllable external device, detects asecond user input (e.g., selection of affordance 2506); and in responseto detecting the second user input, opens an application for controllinga plurality of controllable external devices at the location (e.g.,610-1).

Optionally, at block 2612, the device determines a second position ofthe electronic device within the location; determines that the secondposition corresponds to a second room of the at least one designatedroom, where the second room is different than the first room; and inresponse to determining that the second position corresponds to thesecond room: ceases to display the representation of the firstcontrollable external device associated with the first room; anddisplays a representation of a second controllable external deviceassociated with the second room.

Note that details of the processes described above with respect tomethod 2600 (e.g., FIG. 26) are also applicable in an analogous mannerto the methods described above and below. For example, methods 700,1000, 1300, 1600, 1900, 2100, 2300, 2900, and 3200 optionally includeone or more of the characteristics of the various methods describedabove with reference to method 2600.

In accordance with some embodiments, FIG. 27 shows an exemplaryfunctional block diagram of an electronic device 2700 configured inaccordance with the principles of the various described embodiments. Inaccordance with some embodiments, the functional blocks of electronicdevice 2700 are configured to perform the techniques described above.The functional blocks of the device 2700 are, optionally, implemented byhardware, software, or a combination of hardware and software to carryout the principles of the various described examples. It is understoodby persons of skill in the art that the functional blocks described inFIG. 27 are, optionally, combined or separated into sub-blocks toimplement the principles of the various described examples. Therefore,the description herein optionally supports any possible combination orseparation or further definition of the functional blocks describedherein.

As shown in FIG. 27, an electronic device 2700 includes a display unit2702 configured to display a graphic user interface, optionally, atouch-sensitive surface unit 2704 configured to receive contacts, and aprocessing unit 2706 coupled to the display unit 2702 and, optionally,the touch-sensitive surface unit 2704. In some embodiments, theprocessing unit 2706 includes a determining unit 2708, a displayenabling unit 2710, a causing unit 2712, a detecting unit 2714, and anopening unit 2716.

The processing unit 2706 is configured to: determine (e.g., withdetermining unit 2708) a first position of the electronic device withina location, where the location includes at least one designated room;determine (e.g., with determining unit 2708) that the first positioncorresponds to a first room of the at least one designated room; and inresponse to determining that the first position corresponds to the firstroom, enable display (e.g., with display enabling unit 2710) of arepresentation of a first controllable external device associated withthe first room.

In some embodiments, the processing unit 2706 is further configured to:after a representation of the first controllable external deviceassociated with the first room is displayed, determine (e.g., withdetermining unit 2708) a second position of the electronic device withinthe location; determine (e.g., with determining unit 2708) that thesecond position corresponds to a second room of the at least onedesignated room, where the second room is different than the first room;and in response to determining that the second position corresponds tothe second room: cause (e.g., with causing unit 2712) display of therepresentation of the first controllable external device associated withthe first room to cease; and enable display (e.g., with display enablingunit 2710) of a representation of a second controllable external deviceassociated with the second room. In some embodiments, the representationof the first controllable external device indicates a current state ofthe first controllable external device.

In some embodiments, the processing unit 2706 is further configured to:detect (e.g., with detecting unit 2714) a first user input; and inresponse to the first user input, cause (e.g., with causing unit 2712)sending of instructions to set the first controllable external device toa first state.

In some embodiments, the position of the electronic device is determinedusing GPS, Bluetooth, and/or WiFi signals.

In some embodiments, the processing unit 2706 is further configured to:while the representation of the first controllable external device isdisplayed, detect (e.g., with detecting unit 2714) a second user input;and in response to detecting the second user input, open (e.g., withopening unit 2716) an application for controlling a plurality ofcontrollable external devices at the location.

In some embodiments, the processing unit 2706 is further configured to:enable display (e.g., with display enabling unit 2710) of, based on thedetermination of the room, a representation of a scene profile, wherethe scene profile includes data regarding designated states of aplurality of controllable external devices at the location including atleast one controllable external device associated with the determinedroom.

The operations described above with reference to FIG. 26 are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.27. For example, determining operations 2602, 2604, and 2612, displayingoperations 2606 and 2612, detecting operation 2608 and 2610, sendingoperation 2608, and opening operation 2610 are, optionally, implementedby event sorter 170, event recognizer 180, and event handler 190. Eventmonitor 171 in event sorter 170 detects a contact on touch-sensitivedisplay 112, and event dispatcher module 174 delivers the eventinformation to application 136-1. A respective event recognizer 180 ofapplication 136-1 compares the event information to respective eventdefinitions 186, and determines whether a first contact at a firstlocation on the touch-sensitive surface corresponds to a predefinedevent or sub event, such as activation of an affordance on a userinterface. When a respective predefined event or sub-event is detected,event recognizer 180 activates an event handler 190 associated with thedetection of the event or sub-event. Event handler 190 optionallyutilizes or calls data updater 176 or object updater 177 to update theapplication internal state 192. In some embodiments, event handler 190accesses a respective GUI updater 178 to update what is displayed by theapplication. Similarly, it would be clear to a person having ordinaryskill in the art how other processes can be implemented based on thecomponents depicted in FIGS. 1A-1B.

FIGS. 28A-28E illustrate exemplary user interfaces for providing anelectronic device with access to a location profile, in accordance withsome embodiments. The user interfaces in these figures are used toillustrate the processes described below, including the processes inFIG. 29.

FIG. 28A illustrates user interface screen 2810-1 displayed on device600. User interface screen 2810-1 is associated with a location andincludes affordance 2802 indicating an option to invite a personalcontact to access the location profile of the location.

In response to selection of affordance 2802, device 600 displays a listof personal contacts, as shown in FIG. 28B. Device 600 detects a userinput 2820-1 corresponding to selection of a first personal contact(e.g., Jeffery). After selection of the first personal contact, device600 displays user interface screen 2810-2 as shown in FIG. 28C. Userinterface screen includes affordance 2804 representing an accessparameter and affordances 2806-1 and 2806-2 representing parameters thatfurther specify a user's access. The access parameter determines whethera device or user profile associated with the selected first personalcontact has access to the location profile associated with the location.In some embodiments, access to a location profile permits a user to viewany of the information of the profile (e.g., current status ofaccessories and/or people associated with the location) and to controlaccessories associated with the location. The parameter associated withaffordance 2806-1 determines the editing privileges of the device oruser profile associated with the selected first personal contactincluding, e.g., the ability to edit the states of accessories,automation profiles, and scene profiles. The parameter associated withaffordance 2806-2 determines the remote access privileges of theselected contact.

Device 600 detects a user input corresponding to selection of the accessaffordance 2804 and/or the affordances 2806-1 and 2806-2 and grants anexternal device associated with the first personal contact access to thelocation profile based on the selected parameters. In some embodiments,granting access includes sending instructions to another device (e.g., aserver) to allow the external device associated with the first personalcontact to access the location profile.

Device 600 also sends data including an invitation to access thelocation profile to the external device associated with the firstpersonal contact. FIG. 28D illustrates an external device 2850 thatdisplays an exemplary notification 2808 of the invitation, and FIG. 28Eillustrates an embodiment of an invitation 2812 displayed on externaldevice 2850. Optionally, the invitation is sent either before or afterthe access parameter is selected. In some embodiments, after sending theinvitation to the external device associated with the first personalcontact, device 600 displays an indication that the first personalcontact has been invited to the location. For example, Home screen 610-1in FIG. 6A and/or user interface screen 610-7 in FIG. 6S showing thePeople tab, both described above, optionally include an indication thata personal contact has been invited to the location (e.g., 644-3 “Cory”on user interface screen 610-7 in FIG. 6S).

Optionally, the invitation is sent either before or after granting theexternal device access to the location profile. In some embodiments, theinvitation includes a request to decline or accept the invitation (e.g.,FIG. 28E, affordances 2814 and 2816, respectively). In some embodiments,the invitation is sent before granting access to the location profile,device 600 receives data indicating that the first personal contact hasaccepted the invitation, and then access to the location profile isgranted to the external device associated with the first personalcontact in response to receiving the data indicating that the firstpersonal contact has accepted the invitation. In some embodiments,further in response to receiving the data indicating that the firstpersonal contact has accepted the invitation, device 600 displays anindication of the relative physical location of the external deviceassociated with the first personal contact with respect to the location.For example, Home screen 610-1 and/or user interface screen 610-7 inFIG. 6S showing the People tab, both described above, optionally includean indication that a personal contact is “At home” or “Away”. In someembodiments, a representation of a contact on Home screen 610-1 byitself indicates that the contact is at the location (see also, e.g.,644-2 “Steve” on user interface screen 610-7 in FIG. 6S).

FIG. 29 is a flow diagram illustrating a method for managing access to asystem of controllable external devices using an electronic device inaccordance with some embodiments. Method 2900 is performed at a device(e.g., 100, 300, 500, or 600) with a display and, optionally, atouch-sensitive surface. Some operations in method 2900 are, optionally,combined, the order of some operations is, optionally, changed, and someoperations are, optionally, omitted.

As described below, method 2900 provides an intuitive way for managingaccess to a system of controllable external devices. The method reducesthe cognitive burden on a user for managing access to a system ofcontrollable external devices, thereby creating a more efficienthuman-machine interface. For battery-operated computing devices,enabling a user to manage access to a system of controllable externaldevices faster and more efficiently conserves power and increases thetime between battery charges.

At block 2902, the device displays a first affordance representing afirst personal contact (e.g., FIG. 28B).

At block 2904, the device detects a first user input corresponding toselection of the first affordance representing the first personalcontact (e.g., 2820-1).

At block 2906, the device displays a second affordance representing anaccess parameter that determines access to a location profile associatedwith a location (e.g., 2804), where the location profile includesinformation regarding a controllable external device at the location.

At block 2908, the device detects a second user input corresponding toselection of the second affordance (e.g., selection of affordance 2804).

At block 2910, the device grants an external device associated with thefirst personal contact access to the location profile, where the accessis based on the access parameter.

At block 2912, the device sends data to the external device associatedwith the first personal contact, where the data includes an invitationto access the location profile (e.g., 2808 and 2812). In someembodiments, the invitation is sent before granting access to thelocation profile and access to the location profile is granted to theexternal device associated with the first personal contact in responseto receiving data indicating that the first personal contact hasaccepted the invitation. Optionally, further in response to receivingthe data indicating that the first personal contact has accepted theinvitation, the device displays an indication of the relative physicallocation of the external device associated with the first personalcontact with respect to the location.

Optionally, at block 2914, after sending the invitation to the externaldevice associated with the first personal contact, the device displaysan indication that the first personal contact has been invited to thelocation (e.g., FIG. 6S).

Note that details of the processes described above with respect tomethod 2900 (e.g., FIG. 29) are also applicable in an analogous mannerto the methods described above and below. For example, methods 700,1000, 1300, 1600, 1900, 2100, 2300, 2600, and 3200 optionally includeone or more of the characteristics of the various methods describedabove with reference to method 2900.

In accordance with some embodiments, FIG. 30 shows an exemplaryfunctional block diagram of an electronic device 3000 configured inaccordance with the principles of the various described embodiments. Inaccordance with some embodiments, the functional blocks of electronicdevice 3000 are configured to perform the techniques described above.The functional blocks of the device 3000 are, optionally, implemented byhardware, software, or a combination of hardware and software to carryout the principles of the various described examples. It is understoodby persons of skill in the art that the functional blocks described inFIG. 30 are, optionally, combined or separated into sub-blocks toimplement the principles of the various described examples. Therefore,the description herein optionally supports any possible combination orseparation or further definition of the functional blocks describedherein.

As shown in FIG. 30, an electronic device 3000 includes a display unit3002 configured to display a graphic user interface, optionally, atouch-sensitive surface unit 3004 configured to receive contacts, and aprocessing unit 3006 coupled to the display unit 3002 and, optionally,the touch-sensitive surface unit 3004. In some embodiments, theprocessing unit 3006 includes a display enabling unit 3008, a detectingunit 3010, a causing unit 3012, and a receiving unit 3014.

The processing unit 3006 is configured to: enable display (e.g., withdisplay enabling unit 3008) of a first affordance representing a firstpersonal contact; detect (e.g., with detecting unit 3010) a first userinput corresponding to selection of the first affordance representingthe first personal contact; enable display (e.g., with display enablingunit 3008) of a second affordance representing an access parameter thatdetermines access to a location profile associated with a location,where the location profile includes information regarding a controllableexternal device at the location; detect (e.g., with detecting unit 3010)a second user input corresponding to selection of the second affordance;cause (e.g., with causing unit 3012) an external device associated withthe first personal contact to be granted access to the location profile,where the access is based on the access parameter; and cause (e.g., withcausing unit 3012) sending data to the external device associated withthe first personal contact, the data including an invitation to accessthe location profile.

In some embodiments, the processing unit 3006 is further configured to:after the invitation to the external device associated with the firstpersonal contact is sent, enable display (e.g., with display enablingunit 3008) of an indication that the first personal contact has beeninvited to the location.

In some embodiments, the invitation is sent before granting access tothe location profile, and where the processing unit is furtherconfigured to: receive (e.g., with receiving unit 3014) data indicatingthat the first personal contact has accepted the invitation, whereaccess to the location profile is caused to be granted to the externaldevice associated with the first personal contact in response toreceiving the data indicating that the first personal contact hasaccepted the invitation.

In some embodiments, the processing unit 3006 is further configured to:further in response to receiving the data indicating that the firstpersonal contact has accepted the invitation, enable display (e.g., withdisplay enabling unit 3008) of an indication of the relative physicallocation of the external device associated with the first personalcontact with respect to the location.

The operations described above with reference to FIG. 29 are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.30. For example, displaying operations 2902, 2906, and 2914, detectingoperations 2904 and 2908, granting operation 2910, and sending operation2912 are, optionally, implemented by event sorter 170, event recognizer180, and event handler 190. Event monitor 171 in event sorter 170detects a contact on touch-sensitive display 112, and event dispatchermodule 174 delivers the event information to application 136-1. Arespective event recognizer 180 of application 136-1 compares the eventinformation to respective event definitions 186, and determines whethera first contact at a first location on the touch-sensitive surfacecorresponds to a predefined event or sub event, such as activation of anaffordance on a user interface. When a respective predefined event orsub-event is detected, event recognizer 180 activates an event handler190 associated with the detection of the event or sub-event. Eventhandler 190 optionally utilizes or calls data updater 176 or objectupdater 177 to update the application internal state 192. In someembodiments, event handler 190 accesses a respective GUI updater 178 toupdate what is displayed by the application. Similarly, it would beclear to a person having ordinary skill in the art how other processescan be implemented based on the components depicted in FIGS. 1A-1B.

FIGS. 31A-31B illustrate exemplary user interfaces for customizing thedisplay of a user interface for managing controllable external devices,in accordance with some embodiments. In particular, the user interfacescreens illustrated in FIGS. 31A-31B may allow a user to customize theHome screen (e.g., 610-1 in FIG. 6A) that is presented to the user. Theuser interfaces in these figures are used to illustrate the processesdescribed below, including the processes in FIG. 32.

In some embodiments, device 600 receives data including informationrelated to a location, where the data includes information related to atleast one accessory at the location. In some embodiments, theinformation includes the location profile of a location, which includesinformation such as, e.g., name, address, people, wallpaper, year built,and/or notes, and accessories, scene profiles, automation profilesassociated with the location.

In accordance with receiving the data, device 600 displays an affordancecorresponding to at least one of the accessories at the location. FIG.31A illustrates an exemplary user interface screen 3110-1 displayed inaccordance with receiving the data. User interface screen 3110-1includes a list of accessories associated with a location andaffordances 3102-1 to 3102-5 corresponding to the accessories foroptionally designating the accessories (e.g., as a “FavoriteAccessory”). The accessories are grouped according to rooms.

Device 600 detects a user input corresponding to selection of one ormore of affordances 3102-1 to 3102-5. In response, device 600 displays auser interface screen including an indication of the location and arepresentation of the accessory associated with selected affordance. Forexample, in response to selection of affordance 3102-1, device 600displays a representation of the Accessory 1 under the “FavoriteAccessories” heading of the Home screen 610-1 shown in FIG. 6A, whichalso includes an indication of the location and additional informationsuch as representations of people associated with devices that aredetermined to be currently at the location.

In some embodiments, device 600 displays an affordance corresponding toat least one scene profile associated with the location. FIG. 31Billustrates an exemplary user interface screen 3110-2 including a listof scene profiles and affordances 3102-1 and 3104-2 corresponding to thescene profiles for optionally designating the scene profiles (e.g., as a“Favorite Scene”). Optionally, device 600 detects a user inputcorresponding to selection of one or both of affordances 3102-1 and3104-2. In response, device 600 displays a user interface screenincluding an indication of the location and a representation of thescene profile associated with the selected affordance(s). For example,in response to selection of affordance 3104-2, device 600 displays arepresentation of the Scene 2 under the “Favorite Scenes” heading of theHome screen 610-1 shown in FIG. 6A.

In some embodiments, device 600 receives data representing an invitationto obtain the information related to the location (e.g., FIG. 28E), andthen receives the information related to the location after acceptingthe invitation. For example, in some embodiments, in response toreceiving the data representing the invitation to obtain the informationrelated to the location, device 600 displays the invitation shown inFIG. 28E. Device 600 then receives the information related to thelocation after accepting the invitation (e.g., by selecting affordance2816).

FIG. 32 is a flow diagram illustrating a method for customizing thedisplay of a user interface for managing controllable external devicesusing an electronic device in accordance with some embodiments. Method3200 is performed at a device (e.g., 100, 300, 500, or 600) with adisplay and, optionally, a touch-sensitive surface. Some operations inmethod 3200 are, optionally, combined, the order of some operations is,optionally, changed, and some operations are, optionally, omitted.

As described below, method 3200 provides an intuitive way forcustomizing the display of a user interface for managing controllableexternal devices. The method reduces the cognitive burden on a user formanaging controllable external devices, thereby creating a moreefficient human-machine interface. For battery-operated computingdevices, enabling a user to manage controllable external devices fasterand more efficiently conserves power and increases the time betweenbattery charges.

Optionally, at block 3202, the device receives data representing aninvitation to obtain information related to a location (e.g., 2808 and2812 in FIGS. 28D-28E).

At block 3204, the device receives data including the informationrelated to the location, where the data includes information related toa first controllable external device at the location (e.g., 3110-1).Optionally, the information related to the location is received afteraccepting the invitation.

At block 3206, in accordance with receiving the data, the devicedisplays an affordance corresponding to the first controllable externaldevice (e.g., 3102-1). Optionally, the affordance corresponding to thefirst controllable external device corresponds to a scene profile (e.g.,3104-1), where the scene profile includes data regarding a designatedstate of the first controllable external device.

At block 3208, the device detects a first user input corresponding toselection of the affordance (e.g., selection of affordance 3102-1 to3102-5 or 3104-1 to 3104-2).

At block 3210, in response to detecting the first user input, the devicedisplays a user interface screen including an indication of the locationand a representation of the first controllable external device (e.g.,620-1). Optionally, the representation of the first controllableexternal device on the user interface screen includes a representationof the scene profile (e.g., 640-1). Optionally, the user interfacescreen further includes an indication of the location and arepresentation of a person associated with a remote electronic devicecurrently at the location.

Note that details of the processes described above with respect tomethod 3200 (e.g., FIG. 32) are also applicable in an analogous mannerto the methods described above. For example, methods 700, 1000, 1300,1600, 1900, 2100, 2300, 2600, and 2900 optionally include one or more ofthe characteristics of the various methods described above withreference to method 3200.

In accordance with some embodiments, FIG. 33 shows an exemplaryfunctional block diagram of an electronic device 3300 configured inaccordance with the principles of the various described embodiments. Inaccordance with some embodiments, the functional blocks of electronicdevice 3300 are configured to perform the techniques described above.The functional blocks of the device 3300 are, optionally, implemented byhardware, software, or a combination of hardware and software to carryout the principles of the various described examples. It is understoodby persons of skill in the art that the functional blocks described inFIG. 33 are, optionally, combined or separated into sub-blocks toimplement the principles of the various described examples. Therefore,the description herein optionally supports any possible combination orseparation or further definition of the functional blocks describedherein.

As shown in FIG. 33, an electronic device 3300 includes a display unit3302 configured to display a graphic user interface, optionally, atouch-sensitive surface unit 3304 configured to receive contacts, and aprocessing unit 3306 coupled to the display unit 3302 and, optionally,the touch-sensitive surface unit 3304. In some embodiments, theprocessing unit 3306 includes a receiving unit 3308, a display enablingunit 3310, and a detecting unit 3312.

The processing unit 3306 is configured to: receive (e.g., with receivingunit 3308) data including information related to a location, where thedata includes information related to a first controllable externaldevice at the location; in accordance with receiving the data, display(e.g., with display enabling unit 3310) an affordance corresponding tothe first controllable external device; detect (e.g., with detectingunit 3312) a first user input corresponding to selection of theaffordance; and in response to detecting the first user input, display(e.g., with display enabling unit 3310) a user interface screenincluding an indication of the location and a representation of thefirst controllable external device.

In some embodiments, the affordance corresponding to the firstcontrollable external device corresponds to a scene profile, the sceneprofile includes data regarding a designated state of the firstcontrollable external device, and the representation of the firstcontrollable external device on the user interface screen includes arepresentation of the scene profile.

In some embodiments, the processing unit 3306 is further configured to:receive (e.g., with receiving unit 3308) data representing an invitationto obtain the information related to the location, where the informationrelated to the location is received after accepting the invitation.

In some embodiments, the user interface screen further includes anindication of the location and a representation of a person associatedwith a remote electronic device currently at the location.

The operations described above with reference to FIG. 32 are,optionally, implemented by components depicted in FIGS. 1A-1B or FIG.33. For example, receiving operations 3202 and 3204, displayingoperations 3206 and 3210, and detecting operation 3208 are, optionally,implemented by event sorter 170, event recognizer 180, and event handler190. Event monitor 171 in event sorter 170 detects a contact ontouch-sensitive display 112, and event dispatcher module 174 deliversthe event information to application 136-1. A respective eventrecognizer 180 of application 136-1 compares the event information torespective event definitions 186, and determines whether a first contactat a first location on the touch-sensitive surface corresponds to apredefined event or sub event, such as activation of an affordance on auser interface. When a respective predefined event or sub-event isdetected, event recognizer 180 activates an event handler 190 associatedwith the detection of the event or sub-event. Event handler 190optionally utilizes or calls data updater 176 or object updater 177 toupdate the application internal state 192. In some embodiments, eventhandler 190 accesses a respective GUI updater 178 to update what isdisplayed by the application. Similarly, it would be clear to a personhaving ordinary skill in the art how other processes can be implementedbased on the components depicted in FIGS. 1A-1B.

FIGS. 34A-34K illustrate additional user interfaces for managingcontrollable external devices according to some embodiments. FIG. 34Aillustrates electronic device 3400 having some or all of the features ofdevice 500 described above, including a touch-sensitive display 3402 anda rotatable input mechanism 3404. In some embodiments, electronic device3400 includes some or all of the features of device 100 or device 300,described above. In FIG. 34A, device 3400 displays information relatedto the location profile for location “Ellsworth St.” described abovewith reference to, e.g., FIG. 6A. More specifically, device 3400displays a representation 3406-1 of an accessory associated with thelocation. In some embodiments, representation 3406-1 provides some orall of the features of accessory representations described above, suchas, e.g., representation 620-1 in FIG. 6A. For example, a user canprovide touch inputs on representation 3406-1 to control the state ofAccessory 1. In some embodiments, device 3400 displays only a singlerepresentation of an accessory at a time.

FIG. 34B illustrates device 34 displaying representations 3408-1 and3408-1 of scene profiles. As with accessory representation 3406-1, insome embodiments, representations 3408-1 and 3408-1 provide some or allof the features of scene profile representations described above, suchas, e.g., representation 640-1 in FIG. 6A. In some embodiments, device3400 displays only representations of accessories or representations ofscene profile at a time.

FIG. 34C illustrates one exemplary technique for accessing additionalrepresentations of scene profiles on device 3400. Device 3400 receivesuser input 3410-1 corresponding to a rotation of rotatable inputmechanism 3404. In response to user input 3410-1, device 3400 scrollsthe display to show a portion of a representation 3408-3 of a thirdscene profile. In some embodiments, user input 3410-1 ceases beforerepresentation 3408-3 is completely displayed. As shown in FIG. 34D,device 3400 determines that one or more representations are partiallydisplayed, and in response, continues scrolling the display to show twocomplete scene profile representations, 3408-2 and 3408-3. In someembodiments, device 3400 performs an analogous technique with respect torepresentations of accessories, such that only a single accessoryrepresentation is displayed following user input.

In FIGS. 34E-34F, device 3400 displays a representation 3406-2 of aceiling light accessory. Representation 3406-2 includes an affordance3410. In some embodiments, in response to user input (e.g., a tap) onaffordance 3410, device 3400 displays a control affordance for setting astate of the ceiling light accessory. In some embodiments, the controlaffordance is one of the control affordances described above withreference to FIGS. 6C-6K. In FIG. 34E, the ceiling light is set to 80percent brightness. FIG. 34F illustrates the display of device 3400 whenthe ceiling light accessory is turned off. Compared to FIG. 34E, thedisplay is darker or shaded to indicate that the accessory is in an offstate.

FIG. 34G illustrates a control affordance 3412 corresponding to athermostat accessory. As indicted by the vertical arrows, controlaffordance 3412 can be adjusted to set an upper temperature (e.g., 74degrees) and a lower temperature (e.g., 64 degrees). In someembodiments, similar to control affordance 621 in FIG. 6J, controlaffordance 3412 can be tapped at a location on the affordance to set theupper and lower temperatures. In some embodiments, a user can drag theupper or lower edge of bar 3414 to adjust the upper and lowertemperature, respectively. In some embodiments, device 3400 provides aninterface for selecting the mode of the thermostat, as shown in FIG.34H. In some embodiments, device 3400 switches between the interfacesshown in FIGS. 34G and 34H in response to a horizontal swipe on thedisplay.

FIG. 34I illustrates an embodiment of a notification displayed on device600. As discussed above, in some embodiments, when an accessory isactivated, device 600 displays a notification indicating that theaccessory has been activated and provides affordances for controllingaccessories related to the activated accessory. FIG. 34I illustrates anexemplary notification displayed in response to activation of adoorbell. When the doorbell is rung, device 600 displays notification3416 including an image captured by a camera on the front porch (3416-1)and affordances that can be selected (e.g., via a tap input) to unlockthe door (3416-2), activate an intercom system (3416-3), and turn on theporch light (3416-4). In some embodiments, image 3416-1 is a live videofeed from the camera. In some embodiments, image 3416-1 is a stillimage. In some embodiments, tapping image 3416-1 provides a live videofeed from the camera.

FIG. 34J illustrates an embodiment of a notification 3418 displayed ondevice 3400 in response to the doorbell activation event described withrespect to FIG. 34I. As indicated in FIG. 34J, device 3400 initiallydisplays only an image 3418-1 captured by the camera on the front porch.In response to user input, device 3400 provides affordances forunlocking the door, activating the intercom system, and turning on theporch light analogous to affordances 3416-2 to 3416-4 in notification3416 on device 600. In the illustrated embodiment, in response to userinput (e.g., a swipe on the display or rotation of rotatable inputmechanism 3404), device 3400 displays affordance 3418-2, which can beselected (e.g., via a tap) to unlock the front door. In response tofurther input, device 3400 sequentially displays affordances foractivating the intercom system and turning on the porch light.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the techniques and their practical applications. Othersskilled in the art are thereby enabled to best utilize the techniquesand various embodiments with various modifications as are suited to theparticular use contemplated.

Although the disclosure and examples have been fully described withreference to the accompanying drawings, it is to be noted that variouschanges and modifications will become apparent to those skilled in theart. Such changes and modifications are to be understood as beingincluded within the scope of the disclosure and examples as defined bythe claims.

What is claimed is:
 1. An electronic device, comprising: a display; atouch-sensitive surface; one or more processors; and a memory storingone or more programs, the one or more programs configured to be executedby the one or more processors, the one or more programs includinginstructions for: displaying a first user interface that includes arepresentation of a controllable external device, wherein thecontrollable external device is configured to operate in a plurality ofstates, and wherein the controllable external device is in a first stateof the plurality of states; detecting a first user input correspondingto a selection of the controllable external device; determining whetherthe first user input meets input criteria; in accordance with adetermination that the first user input does not meet the inputcriteria, sending instructions to set the state of the controllableexternal device to a second state of the plurality of states; inaccordance with a determination that the first user input meets theinput criteria: ceasing to display the first user interface; anddisplaying a second user interface that includes a control affordanceindicating the plurality of states of controllable external device andan information affordance; while displaying the second user interface,detecting a second user input on the control affordance corresponding toselection of a third state of the plurality of states; in response todetecting the second user input, sending instructions to set the stateof the controllable external device to the third state; while displayingthe second user interface, detecting a third user input on theinformation affordance; and in response to detecting the third userinput, displaying attributes associated with the controllable externaldevice.
 2. The electronic device of claim 1, wherein the states of theplurality of states are continuous.
 3. The electronic device of claim 1,wherein the states of the plurality of states are discrete.
 4. Theelectronic device of claim 1, wherein the controllable external deviceis a color-adjustable light bulb, and wherein the plurality of statesincludes a plurality of colors.
 5. The electronic device of claim 1,wherein the representation of the controllable external device includesan indication of the current state of the controllable external device.6. The electronic device of claim 1, the one or more programs furtherincluding instructions for: while displaying the second user interface,detecting a fourth user input corresponding to selection of a locationon the display that does not correspond to the control affordance; andin response to detecting the fourth user input, ceasing to display thesecond user interface.
 7. A method, comprising: at an electronic devicewith a display, a touch-sensitive surface, one or more processors, andmemory: displaying a first user interface that includes a representationof a controllable external device, wherein the controllable externaldevice is configured to operate in a plurality of states, and whereinthe controllable external device is in a first state of the plurality ofstates; detecting a first user input corresponding to a selection of thecontrollable external device; determining whether the first user inputmeets input criteria; in accordance with a determination that the firstuser input does not meet the input criteria, sending instructions to setthe state of the controllable external device to a second state of theplurality of states; in accordance with a determination that the firstuser input meets the input criteria: ceasing to display the first userinterface; and displaying a second user interface that includes acontrol affordance indicating the plurality of states of thecontrollable external device and an information affordance; whiledisplaying the second user interface, detecting a second user input onthe control affordance corresponding to selection of a third state ofthe plurality of states; in response to detecting the second user input,sending instructions to set the state of the controllable externaldevice to the third state; while displaying the second user interface,detecting a third user input on the information affordance; and inresponse to detecting the third user input, displaying attributesassociated with the controllable external device.
 8. The method of claim7, wherein the states of the plurality of states are continuous.
 9. Themethod of claim 7, wherein the states of the plurality of states arediscrete.
 10. The method of claim 7, wherein the controllable externaldevice is a color-adjustable light bulb, and wherein the plurality ofstates includes a plurality of colors.
 11. The method of claim 7,wherein the representation of the controllable external device includesan indication of the current state of the controllable external device.12. The method of claim 7, further comprising: while displaying thesecond user interface, detecting a fourth user input corresponding toselection of a location on the display that does not correspond to thecontrol affordance; and in response to detecting the fourth user input,ceasing to display the second user interface.
 13. A non-transitorycomputer-readable storage medium storing one or more programs, the oneor more programs comprising instructions, which when executed by one ormore processors of an electronic device with a display and atouch-sensitive surface, cause the device to: displaying a first userinterface that includes a representation of a controllable externaldevice, wherein the controllable external device is configured tooperate in a plurality of states, and wherein the controllable externaldevice is in a first state of the plurality of states; detecting a firstuser input corresponding to a selection of the controllable externaldevice; determining whether the first user input meets input criteria;in accordance with a determination that the first user input does notmeet the input criteria, send instructions to set the state of thecontrollable external device to a second state of the plurality ofstates; in accordance with a determination that the first user inputmeets the input criteria: ceasing to display the first user interface;and displaying a second user interface that includes a controlaffordance indicating the plurality of states of the controllableexternal device and an information affordance; while displaying thesecond user interface, detecting a second user input on the controlaffordance corresponding to selection of a third state of the pluralityof states; in response to detecting the second user input, sendinstructions to set the state of the controllable external device to thethird state; while displaying the second user interface, detecting athird user input on the information affordance; and in response todetecting the third user input, displaying attributes associated withthe controllable external device.
 14. The non-transitorycomputer-readable storage medium of claim 13, wherein the states of theplurality of states are continuous.
 15. The non-transitorycomputer-readable storage medium of claim 13, wherein the states of theplurality of states are discrete.
 16. The non-transitorycomputer-readable storage medium of claim 13, wherein the controllableexternal device is a color-adjustable light bulb, and wherein theplurality of states includes a plurality of colors.
 17. Thenon-transitory computer-readable storage medium of claim 13, wherein therepresentation of the controllable external device includes anindication of the current state of the controllable external device. 18.The non-transitory computer-readable storage medium of claim 13, the oneor more programs further including instructions that further cause thedevice to: while displaying the second user interface, detect a fourthuser input corresponding to selection of a location on the display thatdoes not correspond to the control affordance; and in response todetecting the fourth user input, ceasing to display the second userinterface.